Subject Profile Analyzing Risk Saves Time for Monitors

To complete rapid, remote source data review (SDR) of critical data and processes for subject assessments, MANA RBM developed, through proprietary processes, a comprehensive, graphical subject profile analyzing risk (SPAR). The SPAR was based on the high risk data elements and processes identified during risk assessment for a PaxVax Phase IV trial implementing risk-based monitoring (RBM). Site monitors used the SPAR to conduct SDR of all subjects enrolled in the trial. This tool was designed to align with the FDA, European Medicines Agency (EMA), and International Conference on Harmonization (ICH) guidance documents on RBM and eSource.^1-5

When compared to onsite SDV, the timesaving using the SPAR was 99.4%. When compared to remote eCRF review, we found a 78% savings in review time of the subject data when using the SPAR (from 8.7 min/subject for the electronic case report form (eCRF) review to 1.9 minutes/subject for the SPAR review). For this study of 900 research subjects, the SPAR provided faster identification of issues and an estimated savings in time of 102 hours of monitoring time. If we compared the SPAR review to onsite SDV, the savings in time for SDV alone would decrease from 6,570 hours for onsite SDV (including travel time) to 28.5 hours for the 900 subjects in the trial. The SPAR is not the only review that is needed, but is the review of the most important data collected for the trial, thus focusing review efforts on the most important aspects of the trial.

The SPAR and eCRF review costs were similar, while the costs of implementing the SPAR was 5% of the costs of traditional onsite SDV. The efficiency of this approach and the immediate availability of the data assured cost-effective, rapid SDR of 100% of subjects.

PaxVax, Inc. and MANA RBM are conducting a year-long research study in 900 research subjects. We incorporated many technology aspects to enhance quality oversight, including direct data entry from EDC; eDiary, including subject-reported dosing; electronic investigator site files (eISF); electronic trial master files (eTMF); and a data repository to enable better oversight of the trial across multiple data sources. The repository included reports, data visualization tools, ad hoc reporting and drill-down capabilities to enable efficient conduct of more study management and oversight remotely.

We also incorporated other aspects of RBM in our remote trial management processes, including risk assessment, changes in timing for data review, SDR, cross-functional team review, issue management and central monitoring review.

As part of implementing RBM and remote site management for the trial, we wanted to synthesize data from the EDC (e.g., medical history [med Hx], concomitant medications [con-meds], adverse events [AE], site RN reviews) and from subject diaries (i.e., 21 days of safety reporting plus dosing information) to provide an efficient method to complete SDR of critical data for subjects.

One of the most difficult parts of monitoring oversight is the “triangular review” of data from AEs, con-meds and medical Hx CRFs. This was further complicated for this study because a key component of the safety oversight required review of data located in two databases.

The subjects reported AEs in the eDiary and a central registered nurse (RN) reviewed the subject reports and eDiary, talked with the subjects and entered safety data into the eCRF, respectively.

High risk data elements that required oversight included: subject dosing (three of four doses were taken at home by the subject); subject reports of symptoms related to vaccine use; subject reports of other changes in health and medication; and the central RN oversight of these subject reports.

SPAR development methods

MANA RBM designed the data collector to be used as eSource. Sites entered the data directly into the collector as they were seeing the subject. This approach enabled the team to see data immediately after a visit without waiting for transcription. It also eliminated any need for SDV and aligned with the FDA guidance on eSource.

All data from the eSource and from the patient reports were imported into a data repository on a daily basis. MANA RBM used existing functionality within the repository to develop a proprietary tool specifically designed to conduct SDR called the Subject Profile Analyzing Risk (SPAR).

The SPAR synthesizes the study-specific, high-risk data elements and processes identified during risk assessment (e.g., subject dosing, subject reporting of safety, all safety assessments) to enable more comprehensive and efficient review of data across multiple data sets and sources. The SPAR design varies depending on the protocol and risk assessment because it includes the critical or high risk data elements and processes.

All the subject and CRF dates were converted to relative days (Day 1 dosing = Relative Day 1) so we could rapidly identify errors in dates. This alone provided a significant advantage to the monitors.

The SPAR, for this study, included data from the following data sets:

• Med Hx (eCRF)

• Con-meds (eCRF and Subject Diary)    

• AEs, including start and stop date; converted to relative day (eCRF)

• Dosing (combined data from eCRF and subject reports)

• Subject report of changes in health and medications (eDiary)

• Subject reports of vaccine symptoms (eDiary)

• Central RN review of subject symptoms (eCRF)

• Central RN required calls to subjects (added in V2)

• End of study (eCRF) added on V2

In addition, we visually positioned eDiary data next to the data from the CRF to make it easier to compare and identify omissions and errors in data. Figure 1 illustrates a portion of the SPAR for a “test” subject. Note that the Relative days are shown at the bottom. In this example, dosing occurred on day 1, 3, 5, and 8, but the protocol required dosing on Days 1, 3, 5, and 7. This resulted in rapid identification of a dosing error because the days that the doses were taken were converted to relative days rather than date; so a quick check to see that the doses were taken on Day 1, 3, 5, and 7 made it is easy to see that one dose was taken on Relative Day 8.

Changes in medication (CM) appear as CM1, 2, and 3 in Figure 1 and represent subject diary reports of changes in medication. The site transferred these findings to the con-med form. Using the SPAR facilitated seeing the CM of sumatriptan was listed, but the start date did not align with the subject report.

Subjects were to report changes in health, which were evaluated by the central RN and entered into the AE form. It is clear to see that the subject reported their health status daily (Event Y/N) throughout the assessment period shown.

It is also easily seen that the site entered the AEs listed by the subject, but the med Hx of migraine raises questions about whether this is a new AE or exacerbation of an existing condition, which can be quickly queried.

Figure 2 illustrates the central RN review of subject-reported known vaccine side effects. For each, the central RN reviewed the reports and made assessments on severity and relatedness. This section of the SPAR allows the monitor to see what reports the subject completed and what reports the central RN reviewed. Missing data are shown as empty boxes.

Review of SPAR vs. direct eCRF
 
To calculate and compare the quality oversight and timesaving of using the SPAR versus direct eCRF review, two monitors, trained in using the SPAR, evaluated 12 subjects using the SPAR and direct monitoring of the eCRF pages. They collected the time required to do each review and recorded their findings.

For instance, when the site monitor used the eCRF to review a subject, this required more than just clicking through the different forms and looking at the data. For example, seeing a con-med was reported on Day 8 of the subject diary would require that the monitor then go to the con-med form to see if it was recorded, then go to the AE page to see if there was an associated AE, making sure the dates of the event or con-med match, querying if necessary, and then going back to where they were at Day 8 to continue the review. The SPAR allowed the monitor to see all those elements of a reported event or con-med at once and know exactly what to query. The SPAR also had “drill-down” capabilities to see exactly what was entered into the EDC or eDiary for any field.

For this experiment, all 21 days of data from a subject’s participation were used.

Time for review results

The average time to complete the review using the SPAR was 1.9 minutes per subject, which included the time to enter queries into the EDC. The calculated time to complete the SPAR review on all 900 subjects would have been 28.5 hours. This activity and the associate time estimate is only one component of remote site monitoring. It supports review of the high risk data elements and processes and did not include all of the other activities that were required of the site monitor. It does illustrate that a review of critical elements of the data for every subject can be performed efficiently and cost effectively.

The average time to review the eCRF data directly was 8.7 minutes per subject. For all 900 subjects, the calculated time for the eCRF review would have been 130.5 hours. For all 12 subjects reviewed, the average number of CRF pages reviewed was 47.5 pages. The total calculated pages for review for the entire study would have been 42,750.
 
Using onsite SDV estimates of 100 pages/day (historical data), the average time to complete SDV for each subject was estimated as 228 minutes and the total time to complete the monitoring only would be 3,420 hours. Estimates of travel time (7 h/trip for one-day trip, historical data) x 450 trips would be 3,150 hours. The time to review all 900 subjects, including travel time, was estimated to be 6,570 hours.

Table 1 shows the various differences in time for review using the different approaches: SPAR, eCRF, and onsite SDV.

Review findings

Slightly more findings were identified using the eCRF review versus using the initial SPAR. Table 2 lists all findings from the monitor’s review using both methods. Two things caused this result. The eCRF review was performed first; many issues were resolved. In addition, upon review of the findings, the SPAR was modified to include additional fields the monitors needed, resulting in similar findings across both methods. SPAR changes were implemented within one month, enhancing the SDR the monitors performed.

The SPAR and our central review process enabled the team to rapidly identify changes that should and were made in training and support materials for the subject to enhance compliance and minimize errors. These changes were implemented within one month.

Training monitors to use the SPAR took only 30 minutes; a significant training timesaving for the quality and safety benefits gained.

Calculated costs

As part of the RBM approach used for the trial, all data from all data sources were imported into a data repository on a daily basis. Reports for overall site performance as well as SDR were developed and used as part of our review process.

To allow accurate comparison of eCRF review versus the use of the SPAR, we listed the cost to develop and amend the SPAR as a separate line item in the analysis below. The cost of the repository was listed for both the SPAR and the eCRF costs since the SPAR was only a part of the overall site performance reports developed for the trial. Table 3 illustrates the time and costs of the three different methods.

Discussion

When the FDA issued its guidance on RBM and eSource in 2013, it emphasized the value of using more technology and immediate remote review to evaluate study conduct based on critical data and processes.

Transcelerate BioPharma has defined SDV as the process of checking data within the study database (e.g., CRF data) against original source data (transcription checking). In contrast, SDR “involves review of source documentation to check quality of source, review protocol compliance, ensure the critical processes and source documentation (e.g. accurate, legible, complete, timely, dated) are adequate, to ascertain investigator involvement and appropriate delegation, and assess compliance to other areas (e.g. SOPs, ICHGCPs).⁶

The SPAR adopts the FDA’s guidance to provide a more comprehensive, integrated view of subject data based on high risk (or critical) data elements and processes identified during risk assessment. It allows monitors to more efficiently and effectively perform SDR. It is not used in isolation, but as an approach to evaluate the subject’s profile comprehensively and in context. It allows monitors to identify areas to further evaluation of the actual data or processes. Monitors use this as the first step in subject review. Following review of the SPAR, they perform a focused check of the eCRF data to identify comments and any other fields that are needed to confirm protocol compliance (e.g., assessors’ initials). They also do remote review of informed consents and all other critical documents for the trial using an electronic investigator site file.

Our monitors called this approach “fun monitoring” because it allowed them to easily identify issues for a subject and to focus on reviewing the subject in context rather than collecting pages of start and stop dates to check AEs and con-meds. This approach also allowed much faster identification of issues.

The SPAR also eliminated the fatigue associated with review of numerous pages of data within a visit and assured the identification of issues that occurred over time and across different data. For instance, an error in dosing was not identified during eCRF review. The date was correct, but the year was wrong. Using the SPAR rapidly identified this error. More importantly, the SPAR allowed all key subject data to be remotely reviewed quickly and comprehensively, providing a total picture of the subject’s participation.

The FDA has emphasized the importance of rapid review of subject data. The SPAR provides an excellent tool to deliver efficient, comprehensive oversight of the high risk data and processes for a trial subject. It also enables review of all the source data for every subject, not just SDV of a subset of the subjects.

The costs savings of 99% were extraordinary when compared to onsite SDV. While the SPAR cost savings were similar to the review of the eCRF data directly for this study, the monitors reported they had a much better grasp of the subject’s data using the SPAR.

An additional benefit of using the SPAR is that it is an excellent tool for investigators to better view their subject’s data than the traditional review of EDC pages.

Summary

The SPAR is an excellent tool for monitors to more efficiently and effectively evaluate subject performance within a trial based on the high risk data elements of trial conduct (i.e., safety, efficacy, protocol compliance). In addition, the SPAR enhances “triangular review” of AEs, con-meds, and med Hx and provides an excellent subject summary for investigators. This tool enables the remote review of all critical subject data integrated in a format that allow far greater understanding of the subject’s participation in the trial.

The SPAR’s cost savings demonstrated the value of using well-planned data visualization tools to provide views of subject data (i.e., SDR) that allow rapid, comprehensive subject review that was highly valued by monitors. The cost effectiveness and remote availability removes the need to do less than 100% review of subject critical data.

Christine McKee, Geraldine Dacpano, and Garrett Manasco are with MANA RBM; Michelle Pallas, Caroline Finlay, Stephanie Galang, and Trudi White are with PaxVax; Richard Erwin is with Catalyst Pharmaceuticals; Lisa Danzig is with PaxVax; and Penelope Manasco is with MANA RBM

References

1. Oversight of Clinical Investigations: A Risk-Based Approach to Monitoring. U.S. Department of HHS, FDA, August 2013 OMB Control No. 0910-0733.
2. Reflection paper on risk based quality management in clinical trials. European Medicines Agency. November 18, 2013 EMA/269011/2013.
3. Electronic Source Data in Clinical Investigations. U.S. Department of Health and Human Services, Food and Drug Administration. Sept 2013. http://www.fda.gov/downloads/drugs/guidancecompliance regulatoryinformation/guidances/ucm328691.pdf
4. Reflection paper on expectations for electronic source data and data transcribed to electronic data collection tools in clinical trials. June 9, 2010. EMA/INS/GCP/454280/2010.
5. ICH, “Integrated addendum to ICH E6(R1): Guideline for Good Clinical Practice E6(R2).” June 11, 2015.
6. Transcelerate BioPharma, Position Paper: Risk-Based Monitoring Methodology. May 20 2013. Transceleratebiopharmainc.com