Centralized Statistical Monitoring As a Way to Improve the Quality of Clinical Data

The last decade has seen the complexity, size, and costs of clinical trials increase, which has made the task of guaranteeing data quality progressively difficult. The number of patients enrolled in randomized trials is steadily growing, while case report forms (CRFs) are becoming more complicated. When combined with the growing trend towards conducting research globally and using sites across multiple continents, these elements are impacting the capability to ensure the quality of clinical trial data, while keeping costs under control. With budgets becoming increasingly difficult to manage and data issues growing by the day, concerns are being raised about whether traditional on-site monitoring methods still stand up as the best solution for examining trial conduct. These approaches rely on extensive on-site visits and source data verification (SDV), which are associated with high costs and a very limited contribution to data quality.¹

The Food and Drug Administration (FDA) and European Medicines Agency (EMA) have both released guidance for the conduct of risk-based monitoring (RBM) techniques that will assist sponsors in better meeting their regulatory obligations.^2,3 RBM enables monitoring activities to be targeted towards those investigative sites that have the best potential to deliver the greatest benefit to a trial. This can be achieved by evaluating the risks to a trial represented by each site and by the data being collected to decide where monitoring efforts would be best placed. However, the lack of appropriate technology and heavy reliance on manual tracking of data are causing challenges in developing successful risk-based monitoring strategies. This is paving the way for new techniques, including central statistical monitoring (CSM), that can accelerate the interpretation of findings related to trial conduct, and ensure the ultimate quality of clinical data.

Despite the number of drug approvals having steadily declined, the cost of clinical research has risen dramatically. If this continues, properly sized randomized clinical trials are likely to become unfeasible. These reductions in clinical trial productivity have also prompted many in the industry to question the value of some of the practices that have become routine in studies and a call for research into more efficient ways of conducting them.⁴ Past research efforts have assessed these practices to identify areas where costs could be reduced without compromising scientific validity. Specific findings indicate that savings could be made by reducing labor-intensive tasks such as on-site monitoring,⁵ which has been shown to represent as much as 30% of the total budget in large global clinical trials.⁶

Current methods of on-site monitoring are largely frequency-based, in line with a prescribed monitoring visit schedule (every 4-10 weeks) aimed at providing quality control at investigative sites. They include significant amounts of SDV to help ensure subject safety and generate quality data. This is at its core a reactive approach, which is limited in its ability to quickly and reliably identify issues and prevent them from recurring. When combined with the current system of regulatory bureaucracy in clinical trials, traditional approaches have led to an extremely expensive research paradigm that, in spite of complex systems of oversight and exhaustive data collection, cannot be shown to adequately ensure the integrity of the research process.⁶ As stated in a recent industry report, “The perception of  “more is better” persists even amid growing concerns that on-site monitoring practices are inadequate to ensure patient safety and data quality.”⁷

Attention has now shifted to pragmatic trial processes that offer improved cost-efficiency, without compromising the quality of the data and the reliability of the trial conclusions. Regulatory preference has been clearly stated by FDA, which “encourages greater reliance on centralized monitoring practices than has been the case historically, with correspondingly less emphasis on on-site monitoring."² EMA has taken a similar stance, stating, "Adaptations to conventional good clinical practice (GCP) methods, for example, adaptation of on-site monitoring visits, sample/focused SDV, new central monitoring processes, etc., subject to appropriate metrics being captured to determine when/if escalation in monitoring would be appropriate."³

The EMA’s position paper on risk based management³ underlines that the quality of a trial needs to be ensured through proper design, while the FDA’s guidance² focuses on strategies for risk-based monitoring. Advances in risk-based approaches and the introduction of new technologies to support these techniques offer the opportunity for sponsors to take a more holistic and proactive approach through off-site and central monitoring, as well as a more targeted approach to on-site monitoring. By incorporating quality and risk management techniques into the design and conduct of clinical trials, risks can be mitigated and issues can be prevented or detected much earlier, improving overall data quality. In line with regulatory requirements, risk assessment should guide clinical trial monitoring plans. The monitoring plan should be developed after the needs and risks associated with a study have been assessed, taking into account the therapeutic area, trial phase and complexity, knowledge of the drugs being used, etc. No single approach to monitoring is appropriate for every trial, hence sponsors should formulate a risk-based monitoring plan early on that is adapted to the risks associated with the experimental procedure. For example, a trial involving innocuous procedures or well-known treatments could involve far less monitoring than a trial involving invasive procedures or experimental new drugs.

The regulatory guidance also suggests that data be analyzed on an ongoing basis to assess and adjust the monitoring strategy as necessary.^2,3 The most widespread way of doing this is via remote monitoring, which involves assessing the data off-site and determining where issues are prevalent. Interestingly, however, a survey published in 2009 under the auspices of the Clinical Trials Transformation Initiative (CTTI) showed that most respondents used centrally available data to assess site performance, but few of them modified the intensity of on-site monitoring based on central findings.⁸

Key risk indicators

To date, RBM approaches have largely relied on key risk indicators (KRIs), which are summary statistics that are pre-defined by the sponsor and potentially reveal deviations in the study conduct, while identifying poor performance in certain centers.⁹ Pre-defined metrics are computed for each site and on-site monitoring frequency can be adapted based on data quality and site performance indicators. For instance, a list of KRIs posted on the website of the British Medicines and Healthcare products Regulatory Agency, includes:¹⁰

• Recruitment rates

• Screen failure rates

• CRF submission/completion times against actual patient’s progress in the trial

• Query rates

• Time to queries resolution vs. number of active queries (site level)

• SAEs reported

• Numbers of missed or late visits/data

• Number of subject withdrawals/dropouts

• Numbers of protocol/GCP non-compliances recorded/reported

• eCRF — audit trail information on completion times in relation to visits or expected timescales

A position paper on risk-based monitoring methodology by TransCelerate BioPharma Inc., a non-profit collaboration consisting of 18 biopharmaceutical member companies, provides further guidance on the use of KRIs.¹¹ Although KRIs are effective to a certain degree, their implementation is far from straightforward. They need to be pre-defined, programmed, tested, and validated, and they only use part of the massive volumes of data collected in clinical trials. In addition, the choice of the thresholds beyond which a KRI needs to be considered as moderately or seriously outside of the norm may be a matter of endless discussions, and may need to be adapted to the characteristics of each trial.

Given the challenges currently faced by the pharmaceutical and biotechnology industry, it may seem strange that statistical theory, which is heavily embedded in the design and analysis of clinical studies, has for so long been overlooked to help optimize risk-based monitoring activities, even though the potential of statistics to uncover fraud in multicenter trials has been given academic attention for a number of years.^12,13,14 More recently, CSM has been highlighted for its potential to take a more neutral view of data, detecting abnormal data, and as such, focus monitoring activities on centers where it is most necessary.^2,3,15

Central statistical monitoring

To effectively target on-site monitoring visits, sponsors may wish to look at their data in numerous ways. By doing so, studies will be significantly de-risked and data quality improved. In comparison to KRIs, which are highly subjective and only indicative of a specific potential for risk, CSM is based on all clinical data. In a full implementation of statistical monitoring, all variables are deemed indicative of quality, from baseline and clinical data to laboratory data, treatment, and patient-reported outcomes; in fact, every bit of data collected is assessed and all variables are considered equally important. In a clinical study, everything collected should be worth collecting and, therefore, worth checking. CSM offers the potential to determine where issues might lie in clinical data during study conduct and before significant problems occur, consequently, helping to avoid any shocks and surprises at the point of regulatory submission. In addition, this method significantly decreases the likelihood for further studies to be performed down the line, should major data issues disqualify or cast doubts on the results of a trial. CSM employs complex statistical algorithms to drill down into individual patient data to detect issues that could compromise the study and jeopardize successful regulatory submission. This approach necessitates minimal work for the sponsor in gaining objective information in order to optimize on-site monitoring by targeting centers at risk. Table 1 compares KRIs and CSM, showing that the two approaches are complementary. The differences shown in Table 1 have direct implications on the nature of the problems identified by each approach.

When CSM is used, variables are typically grouped by CRF section. These are then grouped by visits, visits are grouped by patient, patients are grouped by investigator, investigators grouped by center, centers grouped by country, and countries grouped by geographical region.¹⁶ In the case of a randomized trial, the group allocated by randomization provides another design feature that allows for specific statistical tests to be performed. This is because baseline variables are not expected to differ between the randomized groups, while outcome variables are expected to differ equally in all centers if the treatments under investigation have a true effect. The ability to compare the distribution of all variables in each study site with all other sites means that abnormal patterns can be identified. The approach rests on the premise that the multivariate structure and time dependence of variables in statistical checks are highly sensitive to deviations and extremely difficult to imitate.^12,13,16 Comparisons can be performed with either one variable at a time in a univariate fashion or with several variables, taking into account the multivariate structure of the data, or using longitudinal data when the variable is repeatedly measured over time.^16,17,18 Fabricated data will exhibit abnormal multivariate patterns that are detectable statistically. In addition, humans are known to be poor random-number generators, meaning that tests on randomness can be used to detect data that have been falsified.^12,13

Using a statistical approach necessitates a large number of statistical tests to be performed, including tests on proportions of outliers, means, global variances, within-patient variances, event counts, distributions of categorical variables, proportion of week days, proportion of missing values, and correlations between several variables.^16,17 These tests generate a high-dimensional matrix of P-values, which can be analyzed by statistical methods and bioinformatic tools to identify outlying centers.¹⁶ An additional benefit of CSM is that sponsors who strategically outsource to contract research organizations (CROs) are finding increased efficiencies by using the method as an oversight tool to regularly check the quality of their data. As a result, the solution is also proving essential in helping sponsors select the best sites for future trials.

How the techniques compare

CSM supports RBM by more efficiently detecting errors, sloppiness, tampering, and even fraud, as illustrated in. Regardless of their cause, all these data issues may reveal or constitute a risk to a clinical trial. Although CSM is complementary to KRIs, they are markedly different concepts. KRIs identify centers at risk based on pre-defined variables and known risk factors. As such, this methodology may overlook hard to detect data issues, which may also be indicative of a potential risk. In comparison, CSM does not focus on predefined criteria. Instead, it is agnostic and analyzes all data to detect outlying centers. As a result, the approach is able to detect issues, such as a lack of variability in or implausible values that are unlikely to be detected by other methods.

To successfully implement CSM, a number of important factors must be considered. In the early stages of a trial or in studies that employ numerous small centers, the volume of data available may be too limited to perform statistical tests to detect abnormalities. In addition, as CSM is reliant on computerized data, the technique may miss some types of fraud or errors that can only be detected during site visits, such as evidence provided in hand-written documents or interviews. This considered, on-site visits may be more efficient if monitors perform visits with information about unexpected data patterns identified by statistical monitoring in hand.

To date, there exists little actual evidence that reduced or targeted monitoring methods can achieve the same level of data quality as monitoring with full SDV. However, there is some evidence that most of the findings made during on-site monitoring visits can be detected using CSM methods. Investigators at UK’s Medical Research Council recently reviewed findings made during monitoring visits in a large trial conducted in patients with HIV in Africa.¹⁹ Of 268 monitoring findings, 76 (28%) were also identified in the central database. 179 (67%) could have been identified through central checks, had these been in place, and only 13 (5%) would have required a site visit to be found. Clearly, extensive data checks during on-site monitoring visits are neither cost-effective nor sustainable. In contrast, a CSM approach to quality assurance can yield large costs savings and yet increase the reliability of trial results. What is now required is empirical evidence that CSM can point to issues that would not be identified during monitoring visits.

Conclusion

It is clear that systematically incorporating CSM techniques into the design of a study will lead to tangible benefits and assist sponsors in improving their RBM strategies. By having the capability to identify anomalies in data early, sponsors are provided with the opportunity to address issues as they are uncovered, significantly reducing the risk of regulatory submission failure, while complying with current regulatory guidance and achieving higher quality data at the end of the study. These techniques offer the ability to identify anomalous centers with unusual or suspicious data, while optimizing on-site monitoring and the way in which scarce resources are deployed. CSM represents a major step forward for the industry and the future is certainly bright for those sponsors that embrace an objective approach based on all the data, and nothing but the data.

Marc Buyse, ScD, founder, CluePoints Inc., 185 Alewife Brook Parkway, Suite 410, Cambridge, MA 02138

References

1. Tudur Smith C, Stocken DD, Dunn J, et al. The value of source data verification in a cancer clinical trial. PLOS One 2012; 7: e51623.
2. U.S. Department of Health and Human Services, Food and Drug Administration. Guidance for Industry: Oversight of Clinical Investigations — A Risk-Based Approach to Monitoring. Available at http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM269919.pdf(accessed 20 February 2014).
3. European Medicines Agency. Reflection paper on risk based quality management in clinical trials. EMA/INS/GCP/394194/2011. Available at http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2011/08/WC500110059.pdf(accessed 20 February 2014).
4. Reith C, Landray M, Devereaux PJ, et al. Randomized clinical trials – removing unnecessary obstacles. N Engl J Med 2013; 369: 1061-4.
5. Eisenstein E, Collins R, Cracknell BS, et al. Sensible approaches for reducing clinical trial costs. Clin Trials 2008; 5: 75-84.
6. Califf, RM. ACS and Acute Heart Failure Models. Speaker presentation at the Institute of Medicine Workshop on Transforming Clinical Research in the United States, October 7–8, 2009, Washington DC. Available at http://www.iom.edu/Reports/2010/Transforming-Clinical-Research-in-the-United-States.aspx(accessed 20 February 2014).
7. Cognizant Business Process Services. Risk-based monitoring strategies for improved clinical trial performance. June 2013. Available at
   http://www.cognizant.com/InsightsWhitepapers/Risk-based-Monitoring-Strategies-for-Improved-Clinical-Trial-Performance.pdf(accessed 20 February 2014).
8. Valdes-Marquez E, Hopewell CJ, Landray M and Armitage J. A key risk indicator approach to central statistical monitoring in multicentre clinical trials: method development in the context of an ongoing large-scale randomized trial. Trials 2011;12 (Suppl 1): A135.
9. Morrison BW, Cochran CJ, White JG, et al. Monitoring the quality of conduct of clinical trials: a survey of current practices. Clin Trials 2011; 8: 342-9.
10. Medicines and Healthcare products Regulatory Agency. FAQs for monitoring. February 2013. Available at http://forums.mhra.gov.uk/showthread.php?1675-MHRA-produced-FAQs-for-monitoring/page3 (accessed 20 February 2014).
11. TransCelerateBioPharma Inc. Position paper: risk-based monitoring methodology. May 2013. Available at
    http://www.transceleratebiopharmainc.com/wp-content/uploads/2013/09/Risk-Based-Monitoring-Methodology-Position-Paper.pdf (accessed 20 February 2014).
12. Buyse M, George S, Evans S, et al. The role of biostatistics in detection and treatment of fraud in clinical trials. Statist Med. 1999; 18: 3435-51.
13. Evans S. Statistical aspects of the detection of fraud. In: Lock S, Wells F, Farthing M (Eds). Fraud and Misconduct in Biomedical Research. London: BMJ Publishing Group, 2001 (third edition).
14. Pogue JM, Devereaux PJ, ThorlundK, and Yusuf S.Central statistical monitoring: detecting fraud in clinical trials. Clin Trials 2013; 10: 225-35.
15. Baigent C, Harrel F, Buyse M, et al. Ensuring trial validity by data quality assurance and diversification of monitoring methods. Clin Trials 2008; 5: 49-55.
16. Venet D, Doffagne E, Burzykowski T, et al. A statistical approach to central monitoring of data quality in clinical trials. Clin Trials 2012; 9: 705-13.
17. Kirkwood AA, Cox T and Hackshaw A. Application of methods for central statistical monitoring in clinical trials. Clin Trials 2013; 10: 783-806.
18. Lindblad AS, Manukyan Z, Purohit-Sheth T, et al. Central site monitoring: results from a test of accuracy in identifying trials and sites failing Food and Drug Administration inspection. Clin Trials 2013; Dec. 2.
19. Bakobaki JM, Rauchenberger M, Joffe N, et al. The potential for central monitoring techniques to replace on-site monitoring: findings from an international multi-centre clinical trial. Clin Trials 2012; 9: 257-64.