Impact of IBCTs on Clinical Trial Efficiency

Barriers to entry for implementing an Internet-based clinical trial (IBCT; a.k.a. electronic data capture, or EDC) were described by Kelly and Oldham,¹ and very early on, it became clear what were the obvious advantages and potential impact of IBCTs in drug development. Just a few years later, the emerging roles of Internet technologies for data collection in clinical trials and how IBCTs can and should be effective in collecting clinical data were presented.² The article predicted the eventual common use of IBCTs in the pharmaceutical industry. Richardson³ provided specific details on how to plan and execute an IBCT. Prove⁴ described in detail how Bayer AG had set up and executed a remote site monitoring (RSM) program using a customized laptop.

In a recent survey, 14 pharmaceutical companies were interviewed to determine the best practice requirements for clinical data management and biostatistics.⁵ Among the key metrics were: the rate of case report form (CRF) pages entered and processed, last patient visit to final CRF lock in-house, number of queries generated, and last patient-last visit (LPLV) to database lock. Bart⁶ reported that by using EDC systems, clinical research costs could be reduced by up to 20% due to a reduction in queries, overall reduction in the duration of the clinical trial, and the time to database lock. Hyde⁷ described that one of the benefits of EDC systems is that at the time of data entry, real-time edit checks can be fired to alert for missing and inconsistent data. These edit checks save time and money, and improve the quality of the data compared to the paper query system.

General attributes of EDC have been summarized in a series of publications.^8-12 Time and money are saved by eliminating the need to print and ship CRFs and perform double key entry. When using EDC, double key entry is eliminated because the site enters data directly from the source documents. Because the clinical site personnel enter the data, there are no illegible fields and no one needs to interpret abbreviations or symbols. Costs for query management are also reduced, as there is a marked reduction in the number of queries as well as the time it takes to resolve a query. If a value is out of range, an edit check will prompt the user to check the value at the time of data entry. The role of the CRA has changed with EDC.¹³ The CRA has taken over some of the roles of data management and in part now acts as a quality auditor.

While it is clear that there are many theoretical advantages of an IBCT over a paper-based clinical trial, there is a paucity of data on the efficiencies of: data entry, in-field monitoring, and in-house form review prior to database lock. The present case study uses data derived from an ongoing IBCT using Target e*CRF® in order to evaluate efficiencies of data entry, field monitoring, and in-house data review.

Methods

The data for this case study were derived from an open-label, randomized, multicenter IBCT for the treatment of prostate cancer. Approximately 170 patients participated in this multi-year study. The results represent data collected only during the first year of the study. The goal of the current analysis is to evaluate the efficiencies of IBCTs using metrics derived directly from the IBCT database.

Four forms were selected for analysis: Visit Date, Demographics, Concomitant Medications, and Adverse Events. The roles described below are illustrated and will vary between companies. The analysis evaluated the following:

1. Number of forms that required no modification after initial data entry. This metric represents the quality of data entry by the clinical study site.

2. The time from initial data entry by the clinical study site to the time data are modified. This metric represents both the time that data management or the CRA identifies a monitoring issue and generates a query, as well as how quickly the clinical study site resolves a query.

3. The time from data modification to in-field data review by the CRA. This metric represents the time between query resolution to the time of final source document review by the CRA. At final source document review, the form is designated as "reviewed" by the monitor and the site cannot enter data without the form being electronically opened by the CRA. This metric will vary based on the frequency of monitoring as well as the quality of the monitoring.

4. The time from in-field data review to in-house form review (form "lock"). This metric represents the time between final source document review by the CRA and final acceptance of the form data by the clinical data manager. At this point, the form is designated as "locked" by the clinical data manager and the site cannot enter data without the form being electronically opened by the clinical data manager. Once all forms are locked in-house and all off-line SAS batch validations have been resolved, the patient record can be locked by the data manager.

Results

Efficiencies of data entry. Table 1 summarizes the efficiencies of data entry by the site and efficiencies of in-field monitoring by the CRA. In terms of efficiencies of data entry, 85.9% of all forms did not require any data modification. As expected, Visit Date and Demographics had the fewest changes. The vast majority of the Visit Date (99.3%) and Demographic (96%) forms required no data modifications. In contrast, and as expected, fewer Con Med and Adverse Event forms required no data modifications (71% and 70.2%, respectively). The difference in results between the Visit Date and Demographics forms and the Con Med and Adverse Event forms is that visit dates and demographic data are static and do not change over time, while medication and adverse event data often require updates as new data become available during the course of the clinical trial. Also, the latter two forms are more complex and spelling errors are common.

Table 1. Cumulative frequency (days) from the date of initial data entry to last form modified by the study site.

Table 1 also displays the percentage of forms that were modified within 30, 60, and 90 days of data entry. In general, 76%–99.5% of all forms had their final modification within 30 days of data entry. Having such a high percentage of "clean" forms early in the study cycle allows the CRA to perform source document verification in a very efficient manner.

Efficiencies of monitoring. Table 2 summarizes the efficiencies in data cleaning by measuring the time between final data entry by the clinical study site and in-field source document verification by the CRA. These results will vary based on the frequency of monitoring as well as the quality of the monitoring.

Table 2. Cumulative frequency (days) from the day the form was last modified to CRA review.

In terms of efficiencies of monitoring, 10.1% of all forms were reviewed on the same day the data were modified. What this means is that immediately after the study coordinator modified data (e.g., in response to a query) the monitor was able to review the form. Simultaneous data modification and review occurred 3.7% of the time for Visit Date, 5.4% for Demographics, 30.1% for Con Med, and 17.4% for Adverse Events.

In terms of monitoring efficiencies, the percentage of forms that were reviewed within 30, 60, and 90 days of data entry are also displayed in Table 2. In general, 69%–83% of all forms were reviewed by the CRA in the field within 60 days of data entry or final form modification.

Efficiencies of in-house data reviewers. Rapid source document review also allowed for rapid in-house review of the data (Table 3). The analysis showed that 9.8% of all forms were reviewed by the in-house data reviewers on the same day of CRA review. What this means is that immediately after the CRA verified the source documents, the in-house data reviewer (clinical data manager) was able to "close" the form. Simultaneous form review by the CRA and clinical data manager occurred 9.7% of the time for Visit Date, 3.7% for Demographics, 11.8% for Con Med, and 10.4% for Adverse Events.

Table 3. Cumulative frequency (days) from the date of CRA review to in-house form review.

Table 3 also displays the percentage of forms that were reviewed by the clinical data manager within 30, 60, and 90 days of review by the CRA. In general, 88%–97% of all forms were reviewed by the clinical data manager within 60 days of form review by the CRA.

This rapid data review also allowed for rapid lock of individual patient data by data management.

Discussion

In recent years, with advances in EDC technology and IBCTs, the pharmaceutical industry has made dramatic advances in the area of clinical data management. By using EDC, data entry has been streamlined, data management has become more efficient, and the quality of the data has improved.

The current study allows for the evaluation of some of the preferred metrics described by Fitzmartin.⁵ The data illustrate that there are clear advantages of IBCTs in terms of data entry, source document verification, and data cleanup and management. The types of metrics presented in this article can provide a benchmark to evaluate the operational efficiencies of IBCTs during long-term studies. There is little question that IBCTs can reduce clinical trial costs and improve quality as systems mature and meet the needs of clinical research, data management, and project management.

In contrast to an IBCT, when performing a paper-based clinical trial it is not uncommon for a CRA to discover that the site was not properly prepared for the visit and has not completely filled out the CRFs. The reason for this is that the CRA has no bird's eye view of the data until he/she actually arrives at the study site. Only then does the CRA look at the CRF data and verify that the data entered into the paper CRF match the data in the patient record (source document). The CRA also attempts to reconcile missing, out-of-range, and inconsistent data. As a result, during the monitoring visit the CRA and study coordinator devote more time to resolving issues regarding careless transcription errors, out-of-range data, and missing data than to protocol compliance issues. With EDC, a lot of these issues are resolved prior to the monitoring visit.

One of the main challenges for an IBCT, as well as for a paper-based clinical trial, is to get the clinical study site to enter the data into the CRF in a timely manner. However, once the data are entered, there are major advantages of IBCTs over paper-based trials. With a little effort, any delay in entry/review can be caught up at the end of the trial, enabling database lock one–two days after the last data are available. Often, this only depends on the time it takes to complete the last monitoring visit. This workflow is a major improvement over a paper CRF-based clinical trial, where it can take months to perform final data verification by data management.

The main advantage of an IBCT is the automatic feedback system (i.e., online edit checks), which addresses the most basic errors at the time of data entry (e.g., missing, out of range, and inconsistent data). In paper CRF-based clinical trials, this reconciliation occurs well into the study when data management runs the validation checks after double-key data entry has been completed. In an IBCT, the CRAs are much more knowledgeable about the status of the trial prior to the monitoring visit. From the home office, the CRA can identify and instruct the study coordinator to address missing and illogical data, spelling errors, and incorrect terminologies/acronyms. If there are missing or out-of-range data, the CRA has the explanation, in electronic format, prior to the visit. During the monitoring visit, there is no need for the CRA to make inquiries as to why data are missing, out of range, or inconsistent, since these issues have already been resolved via the edit/logic check and query processes prior to the visit. The CRA can also be authorized to "review" the form electronically to prevent the site from changing data after source document verification is completed. This "review" feature further ensures data integrity.

References

1. M.A. Kelly and J. Oldham, "The Internet and Randomised Controlled Trials," International Journal of Medical Informatics, 47, 91–99 (1997).

2. J. Kuchenbecker, H.B. Dick, K. Schmitz, W. Behrens-Baumann, "Use of Internet Technologies for Data Acquisition in Large Clinical Trials," Telemedicine Journal and e-Health2001, 73–76 (2001).

3. A. Richardson, "Planning and Running the e-Clinical Trial," Applied Clinical Trials, 28–34 (January 2003).

4. J. Prove, "Challenges and Solutions for the Use of Remote Study Monitoring in a Transcontinental Project," Drug Information Journal, 34, 121–127 (2000).

5. R.D. Fitzmartin, "A Survey of Industry Best Practice Metrics in Clinical Data Management and Biostatistics," Drug Information Journal, 1–6 (July/September 2001).

6. B. Thomas, "Comparison of Electronic Data Capture with Paper Data Collection—Is There Really an Advantage?" Business Briefing: Pharmatech, 1–4 (2003).

7. A.W. Hyde, "The Changing Face of Electronic Data Capture: From Remote Data Entry to Direct Data Capture," Drug Information Journal, 32, 1089–1092 (1998).

8. J. Mitchel, J. You, A. Lau, Y.J. Kim, "Paper Versus Web: A Tale of Three Trials," Applied Clinical Trials, 34–35 (August 2000).

9. J. Mitchel, J. You, A. Lau, Y.J. Kim, L. Cheng, S. Fein, R. Nardi, "Clinical Trial Data Integrity. Using Internet-Based Remote Data Entry to Collect Reliable Data," Applied Clinical Trials Supplement, 6–8 (March 2003).

10. J. Mitchel, J. You, Y.J. Kim, A. Lau, B. Levinson, S. Lynch, P. O'Connor, "Internet-Based Clinical Trials—Practical Considerations," Pharmaceutical Development and Regulations, 1, 29–39 (2003).

11. J. Mitchel, C. Ernst, S. Cappi, W. Beasley, A. Lau, Y.J. Kim, J. You, "Implementing Internet-based Clinical Trials," DIA Forum, 40, 22–23 (2004).

12. J. Mitchel, C. Ernst, S. Cappi, W. Beasley, A. Lau, Y. J. Kim, J. You "Meeting the Challenges of Internet-based Clinical Trials," Applied Clinical Trials, 44–53 (June 2004).

13. J. Mitchel, E. Jurewicz, K. Flynn-Fuchs, E. Sharpe, G. Park, "The Role of CRAs in the Development and Implementation of Internet-Based Clinical Trial Applications: New Career Opportunities," Monitor, 17–21 (October 2005).

Jules T. Mitchel,* MBA, PhD, is president, Yong Joong Kim, MS, is senior director of applications development and data management, Joonhyuk Choi, BS, is associate director of application development, and Vanessa Hays, BS, JD, is manager of regulatory affairs with Target Health, Inc., 261 Madison Avenue, 24th Floor, New York, NY 10016, (212) 681-2100, fax (212) 681-2105, email: julesmitchel@targethealth.com Jens Langendorf is datamatician and clinical data manager and Silvana Cappi, MBA, MSc, is executive director, biometrics, with Ferring Pharmaceuticals International, Copenhagen, Denmark.

*To whom all correspondence should be addressed.