Three-Pronged Approach to Optimizing Trial Monitoring

In order to support the transformation of how the pharmaceutical industry manages the performance of clinical trials, in 2013, the Food and Drug Administration (FDA) issued its final "Guidance for Industry: Oversight of Clinical Investigations—A Risk-Based Approach to Monitoring,"¹ and a "Guidance for Industry: Electronic Source Data in Clinical Investigations."² These guidances are consistent with similar recommendations on those topics issued by the European Medicines Agency (EMA): Reflection papers on risk-based quality management in clinical trials³ and expectations for electronic source data and data transcribed to electronic data collection tools in clinical trials.⁴

In 2011, a critical publication on the varied practices of monitoring clinical trials⁵ was published by the Clinical Trials Transformation Initiative (CTTI), a public-private partnership formally established in 2008 by the FDA and Duke University, to identify practices that, through broad adoption, will increase the quality and efficiency of clinical trials. This publication has served as an impetus for the pharmaceutical and device industries, together with regulators, to address the monitoring of clinical trials, which as currently practiced, is inefficient, costly, and, therefore, unsustainable.^6,7 In 2010, a comprehensive paper on the pros and cons of risk-based monitoring (RBM) was published,⁸ with a follow-up paper imploring the industry that it was time to change to RBM.⁹

Quality by design (QbD) is a concept first outlined by Joseph M. Juran and is based on the premise that quality should be part of the project-planning process.¹⁰ According to Juran, most quality crises and problems are due to the way quality is originally planned. While QbD methodologies have been used to advance product and process quality in every industry, they have most recently been adopted by the FDA for drug manufacturing.¹¹ In clinical research, the protocol identifies the quality requirements, and detailed plans and activities complement what is in the protocol. Key factors to QbD methodologies include a well-designed protocol, proper execution of the protocol, steps to assure protocol compliance, corrective and preventative action methodologies, and clear and concise communication strategies.

FDA recently published a paper on QbD methodologies, describing how clinical research is changing, and how FDA and other regulatory authorities are fostering these changes.¹² The eClinical Forum¹³ and TransCelerate BioPharma Inc.¹⁴ have recently published thoughtful papers on how the pharmaceutical and device industries could address RBM. Meanwhile, an approach to quality assurance in the 21st century was published in The Monitor, which described a QbD methodology for clinical research.¹⁵

Results from a Phase II study using RBM and direct data entry (DDE), where the clinical site entered each subject's data into an electronic datacapture (EDC) system at the time of the office visit, demonstrated a major reduction in on-site monitoring compared to comparable studies that use paper source records; EDC edit checks were able to be modified early in the course of the clinical trial, and protocol compliance issues could be identified in real time and rapidly corrected.¹⁶ The use of DDE and near real-time monitoring also led to rapid detection of safety issues. The site reported major cost savings, and estimated that just in terms of data entry, it was able to save 70 hours of labor by not having to transcribe data from paper source records into the EDC system.¹⁷

This article reports the results of the clinical trial initiated in the U.S. and Canada, which included 18 sites and 180 treated subjects; all the sites performed DDE and all the clinical research associates (CRAs) performed RBM, with the bulk of monitoring activities occurring centrally from the home office.

Methods

In addition to a well-designed protocol, the following QbD elements were operationally incorporated into the clinical trial:

Risk-based clinical data monitoring plan (CDMoP): A written strategy was developed to address the review of site-specific source data/documents, the schedule for on-site monitoring, the frequency of central monitoring, and the issuance of central monitoring reports. The CDMoP specified roles and responsibilities as well as the specific monitoring requirements to ensure that the clinical sites complied with the study protocol and regulatory requirements.

The CDMoP also indicated that monitors were to record all monitoring reports in the EDC system, and that all sponsor and study documents were to be maintained in the electronic trial master file (eTMF).

Within the CDMoP, a risk mitigation strategy identified a total of 23 risks to subject safety and/or trial outcome. Each risk was assigned a low to high probability score (1-3) and severity score (1-3). Each risk was then assigned a score, which was a multiple of the two scores, as well as a risk mitigation strategy. For example, "subject dropouts" was one risk to the trial outcome, since any dropout was potentially to be considered a treatment failure. Therefore, subject dropout was assigned a severity score of 3 and a probability score of 2, for a total score of 6. The risk mitigation strategy incorporated was "training and evaluating and resolving reasons for dropouts, phone alerts prompted by the eCRF, and review of online management reports."

DDE: The CDMoP documented that the study would use DDE at the time of the clinic visit. The electronic clinical trial record (eCTR) allowed the study sites to have a contemporaneous electronic copy of the subject's record. To comply with regulations, access to the eCTR was controlled by the clinical investigator or designee and not the pharmaceutical company sponsoring the trial; these original data were stored in "a trusted, third-party repository" prior to the data being transmitted to the EDC database.

QbD meetings: Initially, weekly meetings were held with key team members to review all monitoring activities. Integrated online data management reports addressed safety, quality, compliance, and study-specific issues. As the study progressed, the frequency of these meetings was changed to every two weeks.

On-site and central monitoring activities: As part of the approach to RBM, the CDMoP identified the need to perform both on-site and central monitoring. To avoid the need to duplicate data already within the EDC system, key metrics from the EDC system were displayed within the monitoring reports. The monitoring reports were generated online within the EDC portal and signed electronically by the CRA and the CRA's supervisor. Lists of observations requiring follow-up were also maintained within the EDC portal.

Safety monitoring: A detailed safety monitoring plan was developed. There was nothing unique in this approach to safety monitoring except that an Adobe Acrobat version of an FDA-approved online MedWatch Form 3500A and CIOMS Form 1 could be generated directly from the EDC system for original and follow-up reports. The investigator and the medical monitor could enter online narratives and the medical monitor could control the finalization of the original and follow-up reports needed for regulatory submissions. These reports became an integral part of the EDC system and could be retrieved on demand, based on permissions, anywhere in the world. In addition to the agreed-upon procedures involving serious adverse event (SAE) reporting to the sponsor and regulatory authorities, email alerts occurred at the time of data entry for any SAE and if any SAE data were modified. In addition, the EDC system summarized all adverse events, and it was possible to assess adverse events across sites.

Results

QbD meetings: As part of the QbD methodology, initially, weekly meetings of approximately one hour occurred with the clinical team (n=3), the sponsor (n=2), and an outside expert who performed quality oversight (n=1). Over the course of eight months, this represented a total of 20 meetings and 80 hours (two weeks) of human resources. This effort was roughly equivalent to three on-site monitoring visits.

Time to data entry from the visit date: One of the key advantages never consistently accomplished with EDC was the ability to have rapid access to the clinical trial data from the time of the office visit. With DDE, the site was "forced" to enter the data at the time of the office visit. However, as this involved a change in behavior at the clinical site, there was no guarantee that the sites would comply. Therefore, the time to data entry from the day of the clinic visit was assessed. However, not all data could be entered directly at the time of the office visit since sites maintained certain source records outside the EDC system. As a result, some of the data associated with these source records were entered after the clinic visit. For example, unreported medical histories and medications were identified during "chart review" at the time of the monitoring visit.

In spite of DDE being a "disruptive innovation," 92% of data were entered on the day of the office visit and 95% within five days and 98% within eight days (see Figure 1). Some of the outliers were due to findings during the monitoring visits and delays in data entry when the sites waited for additional information to complete a form.

Time to data review: The time to data review by the monitors is a key factor in optimizing RBM, since without having access to real-time data, the same errors are repeated and any corrective actions are delayed. Key forms included in this analysis were:

• Subject registration

• Demographics

• Medical history

• Visit date

• Vital signs

• Clinical summary

• Drug administration

• Pharmacokinetic (PK) sampling

A total of 13,124 forms were analyzed from 180 subjects (see Figure 2). Results showed that 50% of the forms were reviewed within 13 hours (0.54 days) of data entry, 75% within 27 hours (1.1 days), 95% within 124 hours (5.2 days), and 100% within 335 hours (14 days). It should be noted, however, that occasionally a form was "missed" by the CRA and a small number of forms were "saved" awaiting additional information or conclusions based on consultations with the principal investigator (PI).

On-site and central monitoring activities: Between August 1, 2012 and May 31, 2013, 31 on-site monitoring visits were performed at the 18 sites. No other on-site monitoring was deemed necessary based on the observations at the initial on-site visit, daily review of online eCRFs, in-house review of the eTMF, and site audits by quality assurance. The bulk of the second monitoring visit was combined with the closeout visit since most of the subjects had completed treatment at the time of the visit. The final closeout activities were performed over the phone.

Since measurements on the last day of the three-month treatment phase of the study included evaluation of the primary endpoint, prior to the first subject arriving for that final visit, each site was retrained over the phone as to the required activities taking place on final day of the study. In addition, each site was instructed to inform the CRA when the first subject was to arrive for the Day 90 visit, so that the CRA could immediately review all of the data entered on that day. An email alert was also sent to the project team at the time the Day 90 visit date was entered within the EDC system.

A total of 211 central monitoring reports were issued, and once it was clear that the sites and monitors were adequately trained, the frequency of issuing these reports was changed from every two weeks to every four weeks.

Source data verification: For this study, there were 27,957 EDC "pages" entered for 29 unique CRFs. As part of the approach to RBM, the CDMoP identified specific data elements collected at the sites either within the electronic medical record (EMR) or on paper charts for source data verification (SDV).

A total of 5,581 of these paper/electronic source records were reviewed at the site and compared with the trial database. These records represented about 20% of all entered pages. Results showed that only 13 of the 29 forms had any changes, with a total of 48 changes made to the database as a result of SDV (see Table 1). These changes represented a 0.86% "error rate." The vast majority of the changes (66.6%) occurred in just three forms: medications (13; 27%), medical history (10; 21%), and clinical laboratory result (9; 19%).

In order to evaluate this "0.86% error rate," Table 2 at right identifies examples of types of changes made to the database as a result of SDV. As illustrated, only one modification, titration result (278.3 changed to 123.2), could have had any impact on the study. However, as this parameter was defined as critical to quality (CTQ)—a specific risk to protocol compliance and subject safety—a copy of the record was available to the CRA at the same time the site received it. In addition, all of the changes identified via the SDV process would have had no impact on subject safety, data integrity, or protocol compliance.

Queries: There were 1,099 queries generated from 27,966 CRFs entered by the sites. This represents an overall form query rate of 3.9%. However, only 403 (37.6%) of the queries resulted in changes to the database. Thus, only 1.4% (403/27,966) of forms had database changes as a result of CRA-generated queries.

In order to measure efficiencies of CRA review activities, the time from data entry to query generation was assessed. Strikingly, 39% of queries were generated on the same day as the office visit—58% within one day and 70.6% within five calendar days. What this really means is that corrective actions were able to occur early and rapidly during the clinical trial.

Time from query generation to resolution: Queries were generated in response to an edit check being fired at the time of data entry (auto queries) for which the reason provided by the site required additional information, or as a result of a de novo request for additional information based on clinical review of the CRFs. For example, an ongoing diagnosis of type 2 diabetes was reported but no treatment was documented. While online monitoring was done in real time with current snapshots of data, queries were also generated from online batch edits, which were generated within the EDC system, based on cumulative comparisons of information entered across forms, and over time, that suggested data inconsistencies.

In the previous tables, it was demonstrated that with central monitoring and DDE, it is possible to rapidly enter data and generate queries from the time of data entry. The next challenge was to assure that queries are resolved in a rapid manner. As illustrated in Figure 4, the time to query resolution was assessed for all generated queries, including those done manually, those done based on edit checks being fired at the time of data entry, and those resulting from batch queries run at night within the EDC system.

As shown, with central monitoring, 22% of queries were resolved on the same day they were generated—78% within five calendar days, 91% within 10 days, and 99% within 30 days.

Discussion

One of the keys to a successful outcome of a clinical trial includes timely data entry and data review, and ideally for data entry and data review to occur at the time of the office visit.

Risk has to do with the probability and impact of an event to the outcome of a trial, and risk mitigation strategies are put in place to manage that risk. Clearly, researchers should not put the same effort into monitoring variables that "do not matter" as they do into the ones that "do matter." RBM is not about more or less monitoring visits, or SDV, but rather targeted, efficient, and intelligent monitoring. CRAs need to be retrained in their way of monitoring by focusing on the elimination of errors that matter.

DDE can dramatically reduce or even eliminate paper records, and as a result, SDV should also be dramatically reduced. Since SDV typically assesses how well people transcribe from one medium to another, and since such transcription "error rates" are typically below 1%, SDV as currently performed, should have no impact on the study results. However, as part of the risk assessments performed at the beginning and during the study, the rationale and scope of SDV should be defined. SDV requirements will most likely be replaced, in part, with source data review (SDR) or what would be better described as chart review. Chart review truly allows for a snapshot of the study subject, where critical information "buried" in the chart can be discovered.

After all, monitoring is all about training and oversight. Think about a typical Phase I pharmacokinetic (PK) study. Should the same effort be invested to verify the date of an appendectomy 10 years in the past as would to verify the time of critical PK draws, storage conditions of the samples, and shipping procedures for analysis and methods validation?

The main lessons learned from the study were:

• It is important to identify CTQ measures that had the potential to affect subject safety and protocol compliance at the beginning of the study, to reassess these measures during the study, and train and retrain the clinical sites who are conducting the study and the CRAs who are performing study monitoring and oversight

• When data are entered at the time of the subject's visit (as is done when sites use electronic health records) and data review occurs daily, there will be a marked reduction in the need to perform on-site monitoring

• Traditional SDV by itself adds little value to support data quality and subject safety

• Monitors should now act more like auditors when monitoring clinical sites and require new skill sets and training

• Monitoring of clinical trials may become more of an in-house function than an on-site activity as more data become accessible from centralized monitoring activities

• It is imperative and clearly possible to fundamentally rethink our notions of what it means to monitor clinical trials

The following are recommendations to consider when doing RBM and DDE:

• Accept the fact that errors will occur

• Get the data entered into the database in real time

• Be flexible and make no prior assumptions as to the elements of the RBM plan

• Create a quality section within the protocol to define the topline quality plan

• Have a workable detailed clinical data monitoring plan, including CTQ parameters, a risk assessment and mitigation strategy, monitoring priorities, frequency of data review, and practices to assure that critical procedures and data collection are being done as per the protocol

• Understand that RBM is not about reduced SDV or reduced frequency of monitoring visits, but rather what should be monitored and whether to monitor centrally or on-site

• Train and retrain the sites and monitors on critical issues and variables on a regular basis

• Meet with regulators (e.g., FDA's Office of Scientific Investigations at a Type C meeting or EMA for Scientific Advice) to review the RBM plan and the protocol quality section. This can be done during the early drug development phase, and formal minutes should be generated and maintained

Conclusion

This study clearly demonstrates the advantages of RBM and DDE. Beyond potential cost savings, benefits include:

• Improved timeliness to obtain quality data

• Ability to make faster, mid-course corrections to protocols and EDC systems

• Improved site/sponsor relationships, so both parties can focus on the things that matter and, thus, more effectively allocate resources

• Ability of the sites to see more study subjects during the day since there is virtually no data entry needed once the patient leaves the clinic

• The need for less physical resources at the sites to support on-site monitoring visits

Jules T. Mitchel, MBA, PhD, is President, Target Health, email: [email protected]. Dean Gittleman is Senior Director, Operations, Target Health, email: [email protected]. Judith M. Schloss Markowitz is Senior Project Manager, Target Health, email: [email protected]. Timothy Cho is Associate Director, Application Development, Target Health, email: [email protected]. Yong Joong Kim is Senior Director of Data Management and Application Development, Target Health, email: [email protected]. Joonhyuk Choi is Director of Application Development, Target Health, email: [email protected], Michael R. Hamrell, PhD, is President, MORIAH Consultants, email: [email protected]. Dario Carrara, PhD, is General Manager and Head of Virtual Development, Ferring Galeschines Labor AG, email: [email protected]. Sergio Dalla Nora is Associate Director of Clinical Research, Ferring Canada, email: [email protected].

.— For this publication, Target e*CRF® was used for EDC, Target e*CTR® Viewer was used to access the eSource records, and Target Document was used as the eTMF.

References

1. FDA, August 2013. Guidance for Industry—Oversight of Clinical Investigations—A Risk-Based Approach to Monitoring.

2. FDA, September 2013. Guidance for Industry Electronic Source Data in Clinical Investigations.

3. EMA, 2013. Reflection Paper on Risk-Based Quality Management in Clinical Trials (EMA/INS/GCP/394194/2011).

4. EMA, 2010. Reflection Paper On Expectations for Electronic Source Data and Data Transcribed to Electronic Data Collection Tools in Clinical Trials (EMA/INS/GCP/454280/2010).

5. B. Morrison, C. Cochran, J. Giangrande, et al. 2011. Monitoring the Quality of Conduct of Clinical Trials: A Survey of Current Practices. Clinical Trials, 8:342–349.

6. K. Getz, 2012. Study Monitor Workload High & Varied With Wide Disparity by Global Region, Tufts CSDD Impact Report. January/February, pp. 1-4.

7. K. Getz,, 2012. Flying Blind on CRA Workload, Time Demands. Applied Clinical Trials, July 1, 2012.

8. V. Tantsyura, I Grimes, J. Mitchel, et al. 2010. Risk-Based Source Data Verification Approaches: Pros and Cons. Drug Information Journal 44:745-756.

9. J. Mitchel and J. Schloss-Markowitz. 2011. Risk-Based Monitoring: Time For Change. International Clinical Trials, February: 22-29.

10. M.J.Juran. 1992, Juran on Quality by Design: The New Steps for Planning Quality Into Goods and Services (Free Press).

11. L.X. Yu, 2008, Pharmaceutical Quality by Design: Product and Process Development, Understanding, and Control, Pharmaceutical Research, 25:781-791.

12. L. Ball and A. Meeker-O'Connell. December 2011. Building Quality into Clinical Trials, Monitor pp:11-16.

13. J.M. Brothers, D.A. Gittleman, T. Haag, et al. 2013, Risk-Based Approaches, Applied Clinical Trials, July/August, 26-38.

14. Position Paper: Risk-Based Monitoring Methodology. 2013, TransCelerate BioPharma Inc.

15. J. Mitchel, D. Gittleman, J. Schloss Markowitz, et al. 2013, A 21st Century Approach to QA Oversight of Clinical Trial Performance and Clinical Data Integrity, Monitor, December, 41-46.

16. J. Mitchel, J. Schloss Markowitz, H. Yin, et al. 2012, Lessons Learned From a Direct Data Entry Phase 2 Clinical Trial Under a US Investigational New Drug Application, Drug Information Journal, 46:464-471.

17. J. Mitchel, K. Weingard, J. Schloss Markowitz, et al. 2013, How Direct Data Entry at the Time of the Patient Visit is Transforming Clinical Research—Perspective from the Clinical Trial Research Site, InSite, 2nd Quarter; 40-43.