The Current Status of Risk-Based Monitoring

Risk-based monitoring (RBM) is garnering enormous attention as a change agent in the conduct of clinical trials. This reimagined approach to monitoring promises to reduce the time-consuming and costly practice of on-site 100% source data verification (SDV) while refocusing efforts on improving data quality. Given this capability, use of RBM is being encouraged by the Food and Drug Administration (FDA)¹ as well as the European Medicines Agency (EMA).² To evaluate the current status of RBM, Metrics Champion Consortium (MCC) recently conducted an in-depth survey on the subject.³

This initiative, launched in 2013, was the first detailed study of RBM adoption since the seminal work on RBM conducted by the Clinical Trials Transformation Initiative (CTTI) back in 2009.⁴ That 55-question survey, with 65 respondents representing a range of industry stakeholders, reported that 33% or less of them were using centralized data monitoring to guide, target, or replace site visits. The intervening years have witnessed a sharp rise in acceptance of this methodology across the globe. Specifically, the MCC survey, with its 45 respondents, reported that upwards of 50% were using RBM across programs or on a pilot basis supported by centralized data analytics. Moreover, 10% to 30% of respondents not currently implementing any of the various RBM strategies claimed that they are planning to do so within 12 months. Going forward, RBM adoption is slated to increase further due in part to the growing use of electronic solutions and statistical assessments meant to improve the quality and efficiency of sponsor oversight of clinical trials.¹

The literature on RBM, a methodology using risk algorithms to determine the right level of monitoring5, is increasingly robust.^{6,7,8,9,10,11} This article adds to that trend by detailing some of the results of the survey, which is the first to gather extensive data on specific types of RBM monitoring techniques. Both remote and on-site RBM are discussed, along with insights into the kinds of stakeholders who have chosen to adopt RBM practices and why. Also, results are presented on performance metrics that trigger on-site visits, along with information on the types of e-data sources and data analytics being used.

About the survey

MCC is an association dedicated to the development and adoption of standardized metrics and benchmarks to drive performance improvement in the biopharmaceutical and medical device sectors. A cross-functional team of MCC member companies deployed a 20-question survey to MCC members and non-members alike from June to November 2013. The survey was global in nature, collecting information from North America, Western Europe, Eastern Europe, Asia, Latin America, and “Rest of World.”  Although the final sample was composed of 45 responses representing 40 unique organizations, 70 respondents initially accessed the survey. Of this group, most represented pharmaceutical companies and contract research organizations (CROs) (See Chart 1). 

The survey was deployed between the release of the draft FDA guidance in 2011 and the final FDA guidance in 2013. As defined in the FDA guidance, there is a growing consensus that risk-based approaches to monitoring should focus on risks to the most critical data elements and processes needed to achieve study objectives. Also, RBM approaches are more likely than routine visits to all sites and 100% source data verification to ensure subject protection and study quality.^1,12

Using these definitions from the FDA guidance, the study revealed that all respondents are performing some sort of risk-based monitoring, although the primary reason for adoption varies by type of organization.  Despite this finding, it was not surprising that 85% of the respondents continue employing traditional on-site monitoring practices involving 100% SDV activities (n = 39). What was unexpected, however, was the fact that more than 50% of respondents report they are actively employing some type of RBM program either on a pilot basis or across a full program. The majority of these programs involve on-site monitoring with reduced SDV or remote monitoring, with central data analytics (CDA) supporting some of the efforts. The methods of remote monitoring described in the survey include remote site monitoring using data analytics reports, remote source data verification, and patient profiling, which refers to monitoring individual study subject reports that integrate multiple data points and data sources.

The adoption of RBM varied by phase of clinical trial, with most use of RBM happening in Phase II. Twenty percent of respondents (n = 7) report utilization of RBM in Phase II studies, and 57% (n = 20) claim they are planning to use this methodology in Phase II. Currently, RBM is least likely to be used in Phase III studies, possibly due to the pivotal nature of Phase III studies and reluctance to make the transition with so much riding on the results.* Adoption rates also differed by region, with North America (n = 38) leading the pack, with 55% of respondents reporting currently using some form of RBM, and 39% planning to use it. All regions, however, are expanding their use. It is worth noting that respondents were able to select all responses that applied, so a respondent could report that its company was using RBM in North America as well as Western Europe. (See Chart 2).

Key findings

Results of the MCC study are extensive, generating a 46-page data-filled report. Given the range of respondents and the survey’s global nature, findings fell into several camps, such as who is actively using RBM methods in their clinical trials, types of RBM approaches being implemented, use of CDA, and metrics that trigger on-site monitoring visits. Because of the breadth of the report, this article hones in on a select number of outcomes tied to key reasons for adopting RBM and CDA approaches. Finally, results presented are put in context with previous research.

Adopting RBM

According to the survey, which allowed respondents to make more than one selection, the top three primary reasons for adopting RBM were:

·      Reducing monitoring costs (78% of respondents)

·      Improving data quality (66%)

·      Improving quality oversight (66%)

(See Chart 3)

These percentages are the group mean; when broken down by type of respondent type, the results vary. For CROs and academic research institutes, for example, the key driver for adopting RBM was reducing monitoring costs (92% and 83%, respectively); whereas for pharmaceutical companies, improving quality oversight was the top reason (83%).  For biotech firms, improving data quality was the major reason (83%).

The FDA guidance identifies human subject protection and data quality as the overarching goals for RBM, and although these concerns differ somewhat from the survey findings, generally, they areclosely aligned. Survey results are also similar to the purpose for RBM as stated in the 2013 Reflection Paper from the EMA. The Reflection Paper focuses on the need to overhaul the costly system of oversight to improve quality assurance and quality control in clinical trials and suggests that electronic solutions should be considered (see sidebar). As mentioned earlier, these two factors were among the top three reasons cited in the survey for adoption of RBM.

Central data analytics

As RBM takes hold, it is noteworthy that its growing use is based on capabilities offered by CDA. According to the survey, electronic data sources enable the capture of a wide range of clinical and operational data upon which trend analyses can be performed.  Using this methodology, it is possible to identify outliers remotely along with triggers for conducting on-site monitoring activities. While CDA can support traditional on-site monitoring as well as RBM, the survey highlights anticipated growth in its use in RBM. Specifically, 28% of the respondents indicated that they were planning to use central analytics to support RBM monitoring in the next 12 months compared to only 13% utilizing CDA for traditional on-site monitoring. This trend emerged as a result of the survey defining CDA for both RBM and traditional on-site monitoring.

Data used to develop CDA draw from an array of electronic data sources. Survey results reveal that nine sources are currently being used. As expected, traditional sources such as electronic data capture (EDC) and clinical trial management systems (CTMS) lead the pack, with 66% and 56% of respondents, respectively, already using them. Central lab, imaging, and electrocardiogram (ECG) also have a strong presence, along with interactive voice response systems. The laggard is electronic health records as direct upload and integration of electronic medical and health records is still an emerging science. Details of the various e-data sources appear in Chart 4.

Centralized data may be coming from a whole host of electronic sources, but they serve a common purpose-to improve quality of clinical trials. The survey reveals that the data are used to reach this goal based on different types of CDA methodology, with aggregate data analysis the most prevalent, followed by trend analysis of site characteristics. Currently, 47% of respondents (n = 37) are using aggregate data analysis and 39% plan to begin its use in the next 12 months. Forty-six percent of respondents (n = 38) are using trend analysis of site characteristics, and another 35% claim they plan to start using it in the next 12 months. Other types of CDA, all of which are used much less frequently, include:

·      Statistical probability analysis

·      Statistical determinism

·      Statistical fraud and bias analysis

Detailed explanations of these CDA types are beyond the scope of this article, but in general, each is a different way of evaluating performance by individual sites and aggregated sites. They also use statistical methods and algorithmic models to determine statistical fraud and bias detection. These tools can and are being used in a variety of approaches for remote and on-site RBM. A significant percentage (85%) are using CDA to perform traditional on-site monitoring with 100% SDV, whereas 51% are using it in on-site monitoring with reduced SDV. Nearly half (49%) are using it as part of remote site monitoring, and 26% are using CDA as part of RBM. To determine the range of tools are being used, respondents were able to select all responses that applied.  

Triggers for on-site monitoring visits

On-site monitoring is happening less with the advent of RBM, but it is still necessary periodically. In particular, information gathered through CDA sources can trigger the need for on-site monitoring based on specific performance metrics. The survey asked respondents to identify their most commonly used metrics for triggering on-site monitoring visits. Topping the list was patient enrollment violations (72%), followed by change in principal investigator (71%) (See Chart 5).  

Several of these triggers relate to site staffing issues, such as change in principal investigator, which may reflect the traditional view of monitoring as a means of overseeing site compliance. Going forward, as RBM becomes more established, it will be worth exploring if the top triggers shift to include issues such as delayed entry of data into the case report form or under-reporting of serious adverse events, for example.

In the interim, additional analyses compared these triggers to those reported in the 2009 CTTI monitoring practices survey as well as metrics deemed “critical” in the FDA guidance. Also, an evaluation looked into whether the triggers could be determined via CDA. These analyses found only one area of overlap between the CTTI survey, the FDA guidance, and the top 10 triggers identified in this survey: the number of protocol deviations occurring above a defined threshold.

Among the top five triggers from this survey, only one other-enrollment violations-mapped to the FDA guidance. Interestingly, the top 10 triggers identified in this survey mapped more closely with the 2013 FDA guidance than to the CTTI survey, possibly because the CTTI survey was conducted in Q4 2009, nearly four years before this survey.  Over that time, RBM has become more understood and enjoyed greater acceptance, thereby explaining why this survey’s results were closer to triggers described in the guidance. The fact that the survey was deployed before release of the final guidance yet still had similar findings further underscores the continuing move toward RBM acceptance.

Interpreting the findings

The wealth of information contained in this survey provides a good benchmark for the current status of RBM acceptance. The survey, with 45 respondents representing sponsors, CROs, biotech companies, and academic research institutes, goes beyond providing statistics on current RBM use. It details the percentages of stakeholders planning to implement RBM in the foreseeable future, within 12 months of the survey, and documents that all phases are engaging in RBM, particularly for Phase II studies. 

Opting to embrace RBM reflects keen interest in reducing on-site monitoring costs, which research suggests can represent an estimated 25% to 50% of the clinical trial budget.^13,14 Improving data quality and improving quality oversight, along with other factors are also intense motivators. As part of this effort, stakeholders need to decide what type of RBM they will pursue, such as reduced or tapered SDV, CDA, and on-site monitoring. Our results indicate that stakeholders are using a variety of approaches, depending on the type of study, phase, and willingness to move to RBM.

To some degree, these results coincide with other research on the subject of RBM.  Christison et al report on a survey addressing the industry’s concerns in transitioning away from 100% SDB and toward an RBM approach.¹⁵ With 128 respondents representing large and small pharmaceutical and biotech companies (24%), academic medical institutions (23%), clinical research sites (18%), CROs (14%), medical device companies (8.6%), institutional review boards (3.4%), and other (9%), the reasons they found for adopting RBM were similar to our study’s top finding-reducing monitoring costs. In the Christison study, process improvement was the key driver, with an overwhelming 56% of respondents. There was a sharp drop-off, however, to their second reason-primarily a business initiative-at 16.4%. 

The CTTI survey, with 65 respondents, focused on describing current methods of monitoring and the reasons for their use, but the types of monitoring identified are limited to centralized monitoring alone or as a supplement to on-site monitoring versus always using on-site monitoring. By comparison, our study identifies a whole host of centralized and on-site monitoring options as it relates to RBM, reflecting the continuing advance toward RBM in the few years since the CTTI survey was deployed.

One area where the two studies were similar is the study triggers for on-site monitoring visits. As described earlier, both studies cited patient enrollment violations as the key trigger, and both continued to identify site-related issues, although the nature of those issues differed. Whereas the CTTI study pegged issues of suspected fraud and rate of enrollment as major triggers for on-site monitoring visits, our study focused on problems related to the staffing issues. Moreover, our study went into great detail on the performance metrics that triggered on-site monitoring visits, such as higher than expected randomization/dosing rate, and higher than expected screen failure rate.

Importantly, the survey highlights the continuing march toward greater use of RBM.  Uptake has grown substantially since release of the CTTI survey as suggested by the opinions of two industry veterans (see sidebar). 

Also, the rate of acceptance seems greater than what was reported in CenterWatch back in 2012, which claimed that the industry was slow to implement RBM.¹³ The CenterWatch article cited the CTTI work, and claimed that much of the slowness was linked to fear of change, the amount of upfront work needed to implement RBM, and concern that regulatory bodies may not accept RBM despite the existence of guidance. 

Looking ahead, MCC plans to conduct an annual RBM survey to study the variations and breadth of central and alternative monitoring techniques. At the same time, the MCC’s Risk-Based Monitoring Workgroup is actively working to define processes and metrics, which will enable organizations to implement RBM and evaluate the success of their RBM pilots. As RBM becomes a more routine part of clinical trials, these annual evaluations seek to document its value in improving the clinical trials process. 

Linda B. Sullivan, BS, MBA, is Co-Founder & President of Metrics Champion Consortium, email: lsullivan@metricschampion.org

References

* Additional results available in the full MCC survey. 

1.    Guidance for Industry: Oversight of clinical investigations - A risk-based approach to monitoring. Food and Drug Administration. August 2013. Available at: http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM269919.pdf. Accessed December 20, 2014.

2.    Reflection paper on risk based quality management in clinical trials. European Medicines Agency. November 2013. Available at: file:///C:/Users/Customer/Downloads/EMA%20Reflection%20paper%20risk%20based%20QM%20in%20CT_2013%20(4).pdf. Accessed December 20, 2014.

3.    Taking the Risk: An Examination of Current Organizational Approaches and e-Data Sources Driving Risk-Based Monitoring Programs. Risk Based Monitoring Full Report, 2014.  Metrics Champion Consortium. 2014.

4.    Morrison BW, Cochran CJ, White JG, et al. Monitoring the quality of conduct of clinical trials: a survey of current practices. Clin Trials 2011;8(3):342-9.

5.    Franco R, Hronec M, Karacsony A, et al. Risk-based monitoring: Reduce clinical trial costs while protecting safety and quality. PwC 2013.

6.    Position Paper: Risk-based monitoring methodology. TransCelerate BioPharma. 2013. Available at: http://www.transceleratebiopharmainc.com/wp-content/uploads/2013/10/TransCelerate-RBM-Position-Paper-FINAL-30MAY2013.pdf. Accessed December 11, 2014.  

7.    Koreith K. The high cost and questionable impact of 100% SDV. CenterWatch Monthly 2011;18(2). 

8.    Burgess M. Less is more: Risk-based monitoring of site performance. ICON Insight Report, May 2013. Available at: http://www.iconplc.com/technology/iconik/Less-is-More-Risk-Based-Monitoring-of-Site-Performance.pdf. Accessed December 11, 2014.

9.    Franco P, Hronec M, Karacsony A, et al. Risk based monitoring. Reduce clinical trial costs while protecting safety and quality. PWC. March 2013.

10.Tantsyura V, Grimes I, Mitchel J, et al. Risk-Based source data verification approaches: Pros and cons. Drug Inf J. 2010;44:745-56.

11.Finnigan T, Hall D, Finch C, et al. Position Paper: Designing risk detection indicators for risk-based monitoring. Triumph Research Intelligence, October 2013. Available at: http://www.triumphconsultancy.co.uk/TRI%20Position%20Paper_Designing%20Risk%20Metrics%20for%20RBM_2013-10-22.pdf?cid=actlandingtriumphpositionpaper. Accessed December 19, 2014.

12.Koreith K. The high cost and questionable impact of 100% SDV. CenterWatch Monthly 2011;18(2). 

13.Eisenstein EL, Lemons PW, Tardiff BE, et al. Reducing the costs of Phase III clinical trials. Am Heart J. 2005;149(3):482-8.

14.Redfearn S. Risk-Based monitoring slow to catch on with industry. CenterWatch Monthly, May 2012, Vol 19(5).

15.Christison N, Gilbert KL.Is risk-based monitoring a risky business? Industry perspectives and considerations for implementation. The Monitor, June 2013, pp.19 – 22.