Defining Quality Tolerance Limits and Key Risk Indicators that Detect Risks in a Timely Manner: Reflections from Early Adopters on Emerging Best Practices (Part 1)

Today’s regulatory updates are driving innovation to revolutionize quality improvement in clinical research. While quality tolerance limits (QTLs) are standard in some industries, the International Conference on Harmonization’s ICH E6 (R2)¹ represents the first time QTLs have been introduced for application in good clinical practice (GCP). Although early adopters have collaborated to understand and apply QTL-related practices, most clinical studies employing QTLs have not yet passed through regulatory inspection—leaving much room for discovery of and reflection on emerging best practices.

This three-part series will share lessons learned along the journey to help stakeholders get ahead of the curve for QTL adoption. New insights will help our industry focus not on doing more things but on doing the right things to ensure impactful quality—without overburdening sites or sponsors. These quality improvements represent significant opportunities to reframe and rethink how we bring drugs to market, ultimately offering new hope to patients awaiting medical breakthroughs.

The consortium

The WCG Metrics Champion Consortium(MCC) is part of the WCG Avoca Quality Consortium, an alliance of industry organizations through which sponsors, CROs and vendors collaborate in a progressive, pre-competitive environment. Our shared objective is to enable innovation, elevate quality and bring key stakeholders in the clinical trial process into greater alignment.

In late 2020, consortium members decided to develop guidance on the definition of QTL—regarding metrics, implementation and use—by channeling experiences from across the industry. A working group of 20 individuals (representing 16 organizations) met virtually during 2021 to discuss the challenges and develop a leading practice approach to be shared with the industry. The group adopted and built upon definitions of key terms used by TransCelerate Biopharma Inc. This series summarizes the areas of agreement that the working group considers most beneficial to the industry.

Use of QTLs

Traditionally, quality in clinical trials has been achieved by inspection—checking outputs for errors. Examples include checking data quality before database lock, auditing investigational sites and source document verification (SDV) by site monitors. While other industries have successfully adopted Six Sigma and lean process improvement methodologies—initially developed in the manufacturing industry during the second half of the 20th century—clinical research has been slow to adopt them.

These process improvement methodologies are based on two fundamental premises: designing quality into the protocol and processes from the start (a.k.a. quality by design (QbD)) and monitoring trial processes (rather than their outputs) to maintain them in a consistent state (in control). When a process is determined to be outside the normal operating parameters (out of control), the process is stopped; investigation takes place, and adjustments are made to put the process back on track. By focusing on the process, the intention is that an early signal can be detected—signaling a risk to the outputs. By responding to that signal, any output that does not meet requirements can be minimized and, hopefully, eliminated.

It is preferable not to produce faulty output at all, rather than produce it and then discard or rework it later. Detecting an early signal is critical to this approach. Process improvement efforts often initially focus on understanding the detailed cause and effect in processes to help establish the process measurements most likely to provide a signal that can be detected in time to act. This process focus has spread across many industries—particularly through the adoption of the ISO 9001 standard.²

ICH E6 (R2) was adopted for clinical trials in 2016. Section 5.0 (Quality Management) owes much to the process focus described above. The “predefined QTLs” referred to are measurements that can give an early signal of risk to the overall quality—impacting human subject protection or reliability of trial results. The earlier a signal of risk can be detected, the more likely an investigation can lead to actions that minimize or even eliminate these quality impacts.

Challenges for clinical trials

Consortium members have regularly discussed the challenges of implementing QTLs. One significant challenge is a limited understanding of “cause and effect” within clinical trial processes— in part because every trial is, by definition, new and different. ICH E6 (R2) requires that “important deviations from the predefined quality tolerance limits” are summarized in the clinical study report (CSR). There is, in effect, a balance of judgment taking place (see Figure 1 below). Ideally, an emerging issue is detected early to give enough time to act—to understand the issue and reduce its impact on the trial output. A parameter designed to provide this early signal will lead to signals due to non-systemic issues, i.e., noise.

For example, monitoring the proportion of trial participants who have withdrawn could lead to such a signal if one participant withdraws when only three are enrolled. There may be no evidence that the single withdrawal might impact other participants and, therefore, the trial output. In addition, newly defined parameters designed to detect emerging issues early might be imperfect, leading to further “false” signals.

If a QTL parameter is defined to provide an early signal of an emerging issue, such signals will each have to be assessed for whether they need to be reported in the CSR. Sponsors do not want to have to report deviations that are false signals because of a poorly defined measurement or because they are an artifact of limited data early in the study. As a result, the regulation has the effect of encouraging sponsors to define QTLs that are unlikely to be breached, which undermines the intended purpose of providing early detection of risk.

Figure 1. Balance between providing an early signal with a QTL and the reporting requirements

Another challenge is that the relationship between key risk indicators (KRIs) and QTLs is inconsistent across the industry. Many sponsors have adopted an approach of monitoring investigational site risk using KRIs due to the emergence of risk-based quality management methodologies. Deviations from KRIs do not have to be reported in the CSR, so KRIs can be designed to provide early signals of quality risk without the burden of including a summary of deviations in the final study report. However, the implementation and use of QTLs is often considered a separate exercise. This issue is compounded by the common misconception that any measurement of study-level risk must be a QTL rather than a KRI.

Consortium members participating in the QTL Working Group met over several months in 2021 to discuss these challenges and propose possible solutions. They agreed to accept the definitions of key terms from TransCelerate Biopharma³ and to build on that approach. The following proposals have been made:

• KRIs can be defined and assessed at the study level and other levels (e.g., country, site, participant). QTLs can be considered the most critical study-level KRIs. This concept is described under Quality tolerance limits vs. key risk indicators below.
• Determining the right QTL parameter and threshold for given risks is often difficult. Further guidance can help organizations with QTL definition, including areas where QTLs are typically used and not used. This concept is described in Part 2 of this series.
• The earliest possible detection of an emerging risk is important to managing study risks and reducing the likelihood of them becoming issues that matter. The QTL Working Group proposes the use of companion KRIs to a QTL. These companion KRIs can provide an early signal of an emerging issue but do not lead to the burden of additional reporting in the CSR. The definition of KRIs can significantly impact the timeliness of risk detection. This concept is described in Part 3 of this series.

Quality tolerance limits vs. key risk indicators

Let’s explore the relationship between QTLs and KRIs as applied to clinical research. KRIs are “metrics used to monitor identified risk exposures over time.”⁴ They have a metric definition and a threshold (or possibly two). When the value of the KRI goes beyond the threshold, there is a signal of an emerging risk or issue that may need investigating. They are most often used to detect site-level risks, and they trigger follow-up action steps such as a remote site contact or an on-site monitoring visit to investigate the risks and address any issues. Numerous site-level KRIs may be used in a study, with the total number typically ranging from 10–20. By detecting possible issues at a site early, actions can reduce the potential impact on data integrity or human subject protection.

While sites represent a primary focus of KRI-driven risk detection, the potential sources of risk on a given study may also include the geographic location (e.g., country) or individual participants. Many of the same KRIs used to monitor site risks may also help detect risks at these other levels. Similarly, KRIs can be defined and used at the study level. In this case, the data across all sites are aggregated, and the KRI is compared over time against a discrete risk threshold. Study-level KRIs are thus designed to detect risks and issues that are systemic in nature and not isolated to one or only a handful of sites in the study.

For example, the cycle time from participant visit to data entered into EDC (data entry timeliness) is a commonly used KRI. It detects sites taking significantly longer than expected to enter data, which may pose a risk to data quality. Additionally, it identifies an operational risk that the centralized data monitoring KRI dashboards are not based on near-real-time data. A study-level KRI for data entry timeliness may be useful to monitor the overall trend in the timeliness of data entry across all sites in the study. This could be compared against a standard “expected” time. Suppose the average timeliness across all sites (at study level) is higher than the standard. In that case, there is a risk for the study, which may warrant study-wide actions (e.g., awareness communication to all sites) vs. a site-by-site intervention.

QTLs as a special subset of study-level KRIs

As with KRIs, QTLs are a combination of the parameter (measurement) and threshold level (limit) used to help monitor risks to quality during the conduct of a study. QTLs can therefore be considered a subset of the study-level KRIs, designated to monitor the most important risks associated with critical-to-quality factors⁵ in the study. In other words, they are study-level KRIs that focus on study risks that have the greatest potential impact on data integrity or human subject protection. TransCelerate Biopharma recommends the use of 3-5 QTLs for a study. KRIs are not referenced in GCP—ICH E6 (R2), but QTLs are:

“Predefined quality tolerance limits should be established, taking into consideration the medical and statistical characteristics of the variables as well as the statistical design of the trial, to identify systematic issues that can impact subject safety or reliability of trial results. Detection of deviations from the predefined quality tolerance limits should trigger evaluation to determine if action is needed.”

Perhaps this concept has led to the perspective that QTLs are entirely different from KRIs. Table 1 below helps to highlight the differences and similarities between QTLs and KRIs.

Table 1. Comparison of KRIs and QTLs

According to ICH E6 (R2), QTLs should be “predefined,” which is taken to mean that the parameter and threshold should be established before study start or first participant in (FPI). The QTLs are indicators of whether the expected quality in the trial is being achieved, and, as with hypothesis testing, the expectation is set before gathering data. The aim is to avoid bias from data in the current trial when setting the QTL parameter and threshold. However, this does not mean that a QTL must remain unchanged throughout the course of a trial. Despite best efforts to establish QTL definitions and thresholds in advance, a study team may become aware of information not previously recognized or known that triggers some adjustments. It is important to clearly document the reason and justification for the updates in these cases.

There is an additional requirement in ICH E6 (R2) relating to QTLs:

“The sponsor should describe the quality management approach implemented in the trial and summarize important deviations from the predefined quality tolerance limits and remedial actions taken in the clinical study report (ICH E3, Section 9.6 Data Quality Assurance).”

Compared to KRIs, documentation requirements are significantly different for QTLs, referenced in ICH E6 (R2). The CSR should include an explanation of the selection of QTL parameters and QTL thresholds, a description of actions taken due to important deviations from the thresholds and a justification of any changes to the thresholds.

QTLs are a special subset of KRIs that focus on critical study risks.

Figure 2 below shows the relationship between QTLs and KRIs. The parameters used to monitor QTLs and KRIs are metrics. QTLs are a special subset of KRIs that focus on critical study risks.

Figure 2. Relationship between QTLs and KRIs

There has been much discussion on interpreting the term “important deviation” in ICH E6 (R2). Most deviations from the QTLs are likely to be important, as they are the special subset of KRIs that focus on critical study risks. However, early in the study, where there are few participants, a QTL might be deviated from simply due to limited data. For example, one participant terminating early out of two enrolled may be simply a special cause for that one participant rather than a systemic study-wide issue. These deviations may not be considered “important” as long as the QTL is not exceeded at the end of the study.

Watch for Part 2 in this series which will highlight the process of defining QTLs. If you would like more information on the Consortium, please visit our website.

Authors

Keith Dorricott, MBB, Senior Consultant, WCG Avoca, Steve Young, Chief Scientific Officer, CluePoints, Linda B. Sullivan, MBA (corresponding author), Senior Advisor, Metrics & Performance Management, WCG, Founder and former Executive Director, Metrics Champion Consortium

Contributors to this article include: Maureen Cunningham, United Therapeutics, Adam Czernik, Janssen, Kevin Douglass, Daiichi Sankyo, Todd Johnson, Lokavant, Olgica Klindworth, Medidata (previously associated with PPD), Crupa Kurien, Pfizer, Kim Lombardo, Moderna, Gillian Pilbrow, ICON PLC, Leslie M. Sam, Wool Consulting Group, Kristin Stallcup, Castor, Yumi Sugiura, BMS, Macarena Sahores, Senior RBQM Operations Consultant, TRI

Legal note: Contributions by the co-authors and contributors are solely their own and are not intended to express the views of their organizations.

References

1. ICH E6 (R2) https://www.ich.org/page/efficacy-guidelines
2. ISO 9001 https://www.iso.org/iso-9001-quality-management.html
3. Quality Tolerance Limits: Framework for Successful Implementation in Clinical Development. Bhagat, R et al. Therapeutic Innovation & Regulatory Science. 2020. https://doi.org/10.1007/s43441-020-00209-0
4. TransCelerate Biopharma glossary https://www.transceleratebiopharmainc.com/rbminteractiveguide/the-next-frontier-of-rbm/glossary-key-terms-defined/
5. ICH E8 R1 https://www.ich.org/page/efficacy-guidelines#8-1