Defining Quality Tolerance Limits and Key Risk Indicators that Detect Risks in a Timely Manner: Reflections from Early Adopters on Emerging Best Practices (Part 1)

Today’s regulatory updates are driving innovation to revolutionize quality improvement in clinical research. While quality tolerance limits (QTLs) are standard in some industries, the International Conference on Harmonization’s ICH E6 (R2)¹ represents the first time QTLs have been introduced for application in good clinical practice (GCP). Although early adopters have collaborated to understand and apply QTL-related practices, most clinical studies employing QTLs have not yet passed through regulatory inspection—leaving much room for discovery of and reflection on emerging best practices.

This three-part series will share lessons learned along the journey to help stakeholders get ahead of the curve for QTL adoption. New insights will help our industry focus not on doing more things but on doing the right things to ensure impactful quality—without overburdening sites or sponsors. These quality improvements represent significant opportunities to reframe and rethink how we bring drugs to market, ultimately offering new hope to patients awaiting medical breakthroughs.

The WCG Metrics Champion Consortium(MCC) is part of the WCG Avoca Quality Consortium, an alliance of industry organizations through which sponsors, CROs and vendors collaborate in a progressive, pre-competitive environment. Our shared objective is to enable innovation, elevate quality and bring key stakeholders in the clinical trial process into greater alignment.

In late 2020, consortium members decided to develop guidance on the definition of QTL—regarding metrics, implementation and use—by channeling experiences from across the industry. A working group of 20 individuals (representing 16 organizations) met virtually during 2021 to discuss the challenges and develop a leading practice approach to be shared with the industry. The group adopted and built upon definitions of key terms used by TransCelerate Biopharma Inc. This series summarizes the areas of agreement that the working group considers most beneficial to the industry.

Traditionally, quality in clinical trials has been achieved by inspection—checking outputs for errors. Examples include checking data quality before database lock, auditing investigational sites and source document verification (SDV) by site monitors. While other industries have successfully adopted Six Sigma and lean process improvement methodologies—initially developed in the manufacturing industry during the second half of the 20th century—clinical research has been slow to adopt them.

These process improvement methodologies are based on two fundamental premises: designing quality into the protocol and processes from the start (a.k.a. quality by design (QbD)) and monitoring trial processes (rather than their outputs) to maintain them in a consistent state (in control). When a process is determined to be outside the normal operating parameters (out of control), the process is stopped; investigation takes place, and adjustments are made to put the process back on track. By focusing on the process, the intention is that an early signal can be detected—signaling a risk to the outputs. By responding to that signal, any output that does not meet requirements can be minimized and, hopefully, eliminated.

It is preferable not to produce faulty output at all, rather than produce it and then discard or rework it later. Detecting an early signal is critical to this approach. Process improvement efforts often initially focus on understanding the detailed cause and effect in processes to help establish the process measurements most likely to provide a signal that can be detected in time to act. This process focus has spread across many industries—particularly through the adoption of the ISO 9001 standard.²

ICH E6 (R2) was adopted for clinical trials in 2016. Section 5.0 (Quality Management) owes much to the process focus described above. The “predefined QTLs” referred to are measurements that can give an early signal of risk to the overall quality—impacting human subject protection or reliability of trial results. The earlier a signal of risk can be detected, the more likely an investigation can lead to actions that minimize or even eliminate these quality impacts.

Consortium members have regularly discussed the challenges of implementing QTLs. One significant challenge is a limited understanding of “cause and effect” within clinical trial processes— in part because every trial is, by definition, new and different. ICH E6 (R2) requires that “important deviations from the predefined quality tolerance limits” are summarized in the clinical study report (CSR). There is, in effect, a balance of judgment taking place (see Figure 1 below). Ideally, an emerging issue is detected early to give enough time to act—to understand the issue and reduce its impact on the trial output. A parameter designed to provide this early signal will lead to signals due to non-systemic issues, i.e., noise.

For example, monitoring the proportion of trial participants who have withdrawn could lead to such a signal if one participant withdraws when only three are enrolled. There may be no evidence that the single withdrawal might impact other participants and, therefore, the trial output. In addition, newly defined parameters designed to detect emerging issues early might be imperfect, leading to further “false” signals.

If a QTL parameter is defined to provide an early signal of an emerging issue, such signals will each have to be assessed for whether they need to be reported in the CSR. Sponsors do not want to have to report deviations that are false signals because of a poorly defined measurement or because they are an artifact of limited data early in the study. As a result, the regulation has the effect of encouraging sponsors to define QTLs that are unlikely to be breached, which undermines the intended purpose of providing early detection of risk.

Another challenge is that the relationship between key risk indicators (KRIs) and QTLs is inconsistent across the industry. Many sponsors have adopted an approach of monitoring investigational site risk using KRIs due to the emergence of risk-based quality management methodologies. Deviations from KRIs do not have to be reported in the CSR, so KRIs can be designed to provide early signals of quality risk without the burden of including a summary of deviations in the final study report. However, the implementation and use of QTLs is often considered a separate exercise. This issue is compounded by the common misconception that any measurement of study-level risk must be a QTL rather than a KRI.

Consortium members participating in the QTL Working Group met over several months in 2021 to discuss these challenges and propose possible solutions. They agreed to accept the definitions of key terms from TransCelerate Biopharma³ and to build on that approach. The following proposals have been made: