The Role of KRIs in Process-Driven RBQM Success

Using key risk indicators (KRIs) can improve clinical trial data quality in 80% of the sites on average.¹

These were among the findings of a new study, thought to be the first to quantitatively assess how KRIs or centralized monitoring metrics that act as indicators of risk in specific areas of study conduct can improve trial quality.

“These results provide clear evidence that centralized monitoring, recommended by regulatory agencies, is effective at detecting and resolving data quality issues in clinical trials,” said the authors.

“When properly implemented, managed, and followed up, KRIs enable a more targeted and efficient approach to identifying and addressing emerging quality-related risks during a study.”

Centralized monitoring and KRIs: The story so far

Historically, sponsors and contract research organizations (CROs) have relied heavily on source data verification (SDV) to ensure clinical trial participant safety and data quality. However, this process, which sends site monitors on frequent visits to trial sites to review source data, has its challenges.

First, it is labor-intensive and prone to human error, making it inefficient and potentially unreliable. In addition, the advent of hybrid and decentralized clinical trials (DCTs) and remote patient monitoring through devices, including wearables, has significantly expanded the volume of data collected away from the site. This is further eroding the relevance of SDV as a quality oversight method.

Enter centralized monitoring, which uses data analytics to detect emerging quality-related risks during a clinical trial.

The approach enables study teams to investigate and then take any necessary corrective actions before an identified issue has a chance to threaten patient safety or data reliability. This leads to safer, more efficient clinical trials and fewer delays to market access.

While first encouraged in a 2016 update to the International Committee on Harmonization of Good Clinical Practice (ICH-GCP) guidelines, it was during the height of the COVID-19 pandemic that this method, which is a critical operational component of RBQM, really took hold.

Authorities, keen to avoid wide-scale study shutdowns in the face of travel restrictions and social distancing regulations, recommended sponsors and CROs replace on-site monitoring with the centralized approach.

The case for KRIs

Centralized monitoring involves the regular review of study data from various sources. These sources can include electronic case report forms (eCRFs), central laboratories, electronic patient-reported outcomes (ePRO) and electronic clinical outcome assessment (eCOA) systems, wearable technologies, and clinical trial management systems (CTMS).

It uses various analytics tools to search this disparate data for patterns and trends that could indicate quality or safety concerns. One of the most commonly used centralized monitoring tools includes statistical data monitoring (SDM) and key risk indicators (KRIs) - which can also include quality tolerance limits (QTLs).

Typically, at the start of a study, the study team defines a set of KRIs to monitor targeted risk areas. Common KRI examples include non-serious AE reporting rate (AERATE) and eCRF visit-to-entry cycle time (V2ECT). Then, each of the KRIs defines risk alert thresholds that can be discrete values or set dynamically based on a statistical comparison of each site’s KRI value with the trend across all sites in the study. If, upon regular data review, a site deviates from these thresholds, a “risk signal” is generated, prompting the study team to investigate.

To add to the evidence base and to understand how KRIs can improve trial data quality, the analysis conducted by CluePoints evaluated the effectiveness of nine commonly used KRIs in 212 studies from across 23 sponsors implemented on the company’s RBQM platform.

The selected KRIs, which included non-Serious AE reporting rate (AERATE), protocol deviation rate, eCRF visit-to-entry cycle time (V2ECT), and early termination rate, monitored a wide range of clinical and operational risks across the realms of safety, compliance, data quality, enrollment, and retention.

In total, 1,676 sites with KRI risk signals were included in the analyses to evaluate the tools’ ability to enable timely corrective action that brought site KRI-related behaviors back into an expected range during the study.

Across all KRIs, quality improvement was observed in 82.9% of the KRIs (1,680/2,027) as measured by statistical score and 81.1% (1,467/1,809) as measured by the observed KRI value.

For those risk signals in which improvement was observed, the sites’ KRI statistical scores improved an average of 66.1%, closer to the expected level when the risk signal was closed. For observed KRI values, the average level of improvement was 72.4% toward the expected value.

While the analyses recorded improvements across all selected KRIs, there were some standout results. For V2ECT, there was a statistical score improvement at 92.1% of sites (467/507) and an observed value improvement at 84.7% of sites (411/485). Additionally, those sites for which V2ECT improved almost completely closed the gap toward the expected value.

Process-driven success

The analysis also demonstrated that centralized monitoring success is highly dependent on the thoughtful design and implementation of all RBQM tools and risk follow-up processes.

This was shown in the journey of a risk signal generated when one site’s average entry cycle time (V2ECT) reached 32 days, significantly higher than the study average of five days (Figure 1A).

Figure 1 A and B. A V2ECT and B AERATE signal examples

Following intervention by the clinical research associate (CRA) to discuss the delayed data entry with site personnel, the average V2ECT fell to just four days. This was slightly better than the five-day average observed across all sites in the study—and thus represented an improvement of more than 100% relative to the expected value.

In addition, 79.5% of sites (58/73) improved their AE reporting rate (AERATE) following study team intervention. Interestingly the analysis found that for most of the signals, an investigation by the CRA could not confirm any issues in the AE collection and reporting process. Nevertheless, it was evident that discussions between CRAs and site staff raised awareness and resulted in improved compliance.

One oncology trial, for example, had an observed study-wide average AERATE of 0.8 AEs per patient visit, while one center had an observed AERATE of zero across 16 patient visits (Figure 1B). The expected number of AEs for this site was 12.8, given the study-wide trend. This generated a risk signal that resulted in site re-training on AE collection and reporting expectations by the CRA.

Subsequently, the site started to report AEs, some of which had a start date older than the signal creation date, confirming they had not been reported as expected before the training. When the risk signal was closed five months later, the site had reported a total of 12 AEs across 19 patient visits. It continued to improve its AE reporting behavior through the completion of the study.

Future of centralized monitoring

Centralized monitoring and RBQM can help sponsors, currently faced with the ever-increasing clinical trial complexity, costs, and timelines, reap the rewards of safer, more efficient, and more reliable drug development.

As regulation and business need to move forward, organizations are looking for practical ways to embed risk-based approaches into their everyday workflows—and, as this paper shows, using KRIs as part of a managed quality management program can help them do just that.

Steve Young, chief scientific officer, and Sylviane de Viron, data and knowledge manager; both with CluePoints

Reference

1. de Viron, S., Trotta, L., Steijn, W., Young, S., & Buyse, M. (2022). Does Central Monitoring Lead to Higher Quality? An Analysis of Key Risk Indicator Outcomes. Therapeutic Innovation & Regulatory Science, 1-9.