De-Risking Trials with Science-Driven Oversight

In 2013, the FDA and EMA released final guidance to enhance monitoring by focusing monitoring on the highest risk areas of a trial–a risk-based approach. They recommended use of a risk assessment approach to identify those areas of highest risk. The guidance was finalized by the ICH and adopted by the EMA in 2017, and the FDA in 2018.

MANA RBM is a clinical trials institute and research laboratory. We design, test, and publish methods for conducting remote trial management for traditional site-based trials or decentralized trials. MANA RBM uses an approach called Science Driven Oversight, which is based on using protocol-specific analytics that enable cross-database, automated identification of “errors that matter” and systematic errors which are not readily detected by standard review or SDV.

Determining risk

In 2019, at the start of a large, complex oncology trial, MANA RBM conducted an assessment to determine the areas of highest risk for the trial. Our risk assessment is focused on the “Significant Six” or the six areas of trial conduct that absolutely have to be right:

1. Informed Consent collection and processes
2. Primary Efficacy data collection and processes
3. Safety: identification of critical safety issues and recognizing safety signals
4. Investigational Product Management
5. Protocol Compliance
6. Critical Document collection, including subject source documents.

After identifying the most significant risks as protocol-specific “Errors that Matter”, data and document collection systems were designed and built that enabled us to gather the relevant information to identify errors with the highest risk scores.

Protocol-specific analytics were developed to better identify whether these significant risks occurred. Our goal was to not only identify whether the Errors that Matter occurred, but to proactively prevent the errors using our automated analytic tool, REACHER.

For this trial, four of the largest areas of risk for the trial were as follows:

• Subjects may be enrolled in the study that do not meet the complex Inclusion/Exclusion Criteria.
• Processes for primary and key secondary endpoints must be correct
• Subjects may experience a SUSAR (Suspected, Unexpected, Serious Adverse event) that we would not recognize, resulting in missed reporting requirements
• Unblinded IP processes may result in errors affecting the integrity of the trial

Following our initial assessment, major changes were made in the protocol. After finalizing the protocol and re-evaluating the risk assessments, including taking into account the automated protocol- specific analytics, we were able to significantly decrease the most critical risks. The large decreases in the risk scores were due, primarily, to the increased ability to detect the errors.

For new risks not identified in the original risk assessment, the total risk score was lower than would have been the case had we not had a high ability to detect the errors. In one instance the risk could not be decreased because of constraints imposed by the reporting tools available from the IP vendors. Their reporting limitations decreased the ability of the analytics and our team to detect when an error occurred.

Table 1 below illustrates a subset of the risks identified during the Risk Assessment done in 2019 and 2020. Note that a low ability to detect increased the risk score by a factor of five compared to a high ability to detect.

Table 1. Subset of Risk Assessment for Phase III Oncology Trial

Discussion

This exercise illustrates several important aspects of our risk-based monitoring approach.

First, the risk score can be positively impacted (reduced) by proactively developing protocol-specific analytics to identify whether errors occur in the areas of highest risk. This is illustrated in our example of the analytics surrounding subject eligibility for randomization. By designing an analytic report to assess all of the available underlying data prior to randomization, the potential risk of incorrectly randomizing an ineligible subject can be identified quickly. In fact, this error can be eliminated entirely and one of the most critical risks to subject safety can be prevented.

Second, the risk assessment should be conducted at multiple times during a trial to assure that any process changes are captured . High risk items should have specific data collection design and an associated report or analytic tool to assure that oversight is focused on the most critical aspects of the trial.

Third, in conducting oversight, the challenge is often getting the sites to enter the data promptly for rapid assessment of errors that matter. This is a major change in standard working practices, where there is often a separate person that completes data entry. That is why the FDA included an eSource guidance document finalized in 2013. Incorporating the prompt data entry requirement into contracts, repeated reinforcement, and demonstrating to the site the value of proactive data analysis and rapid feedback will all help to change this engrained site process.

Finally, this paper also illustrates that pre-existing company limitations related to critical data exports, as we encountered here with IP management systems, can have deleterious effects on the ability to detect some of the most important errors associated with a trial. Companies that provide IP services should have robust reporting and notification tools that provide data across all research sites available to help assure that errors can be eliminated (or caught before they impact a subject’s treatment). Older, “established” companies may not have kept up with the evolving area of data exports, which can have a negative impact on risk-based oversight strategies. Evaluation of vendors should include examination of this critical capability to prevent unwelcome oversight surprises and the inability to mitigate risk.

Summary

Risk assessments focused on the significant six areas of highest study and subject risk are a valuable tool to develop protocol specific analytics and review processes for oversight. By focusing risk assessment in these six areas and tying the highest risk areas with reports that are routinely reviewed can significantly decrease the risks for a trial. Using near real time analytic tools offers a proactive solution to prevent errors before they occur and deleteriously affect subject safety or trial data integrity.

Mavys Palacios is a Consultant; Livia Tran is an Analytics Team Programmer; Kevin Fielman is an Analytics Team Leader; Charles T. Lancaster is a Project Manager; and Penelope Manasco is the CEO, all of MANA RBM