Standardized Metrics Capture Most, Not All Clinical Trial Risks

January 5, 2017

Article

The standardization of trial metrics offers numerous benefits towards overseeing clinical trials, optimizing clinical operations and mitigating study risks. However, on a situational basis customized metrics may become necessary to address study-specific risks, uncover unknown issues and demonstrate results.

Biopharmaceutical enterprises are currently at a stage, where they are leveraging technology systems in order to aggregate data into an accessible repository, and enabling efficient key risk and performance indicators (KRIs/KPIs) implementation and visualization in clinical trials. These advances have facilitated the adoption of standardized industry metrics, such as those offered by the Metrics Champion Consortium (MCC). While standardized measurements and tools offer momentous benefits to the industry as a whole (as it relates to clinical trial oversight, risk management, and performance), the omission of study-specific KPIs/KRIs could also result in missing critical study issues. Moreover, depending solely on KPIs/KRIs without incorporating mitigative business operational processes can result in retroactive, rather than proactive decision-making and risk management.

In this article, we will investigate the applications of standardized metrics, examine situations where study-specific indicators can be applied, understand the incorporation of risk-mitigative business operational processes, and consider the importance of uncovering unknown risks during study operational execution.

Standardized clinical trial metrics

The rise of standardized measurement analytics in the clinical trials industry is a notable advancement that has transcended clinical trial operations and risk management into modern industrial standards. A recent industry survey has demonstrated that an average of 83% of biopharmaceutical enterprises currently use standardized performance metrics in both clinical operations and data management functions, and nearly 70% of enterprises use such metrics in quality management¹. MCC’s analytics have expanded from the very basics of clinical operational performance to therapeutic-specific clinical study aspects (Figure 1).

Figure 1: MCC KPI/KRI Categories⁶

Standardized metrics offer a comprehensive overview of clinical trial operations, which can be used to enhance oversight and facilitate decision-making in order to optimize clinical trial performance and improve quality.

Do standardized metrics cover it all?

While the standardization of clinical trial metrics ameliorates study oversight and measurement, they may not necessarily capture all critical study risks. Factors, such as therapeutic indication, study design, drug or device, and study phase introduce variability in risk and performance measurement that may not be captured by standardized metrics.

For example, the endpoints for an ultrasound medical device may depend on the quality of recorded ultrasound waves and the skills of a qualified site staff member. Therefore, the quality management system must quantify device-specific KPIs/KRIs to diagnose device-related issues, as well as incorporate analytical adjudication processes for objective ultrasound recording review. Thus, the quality risk management plan drives action-the additional training to staff members who quantitatively demonstrate poor ultrasound recording quality.

KPIs/KRIs in combination with business processes

The Plan, Do, Check, Act approach suggests that planning the identification and monitoring of issues, acting on them when they occur, ensuring they have been resolved, and measuring the improvement of the identified issue² contribute towards a comprehensive quality management system. While standardized clinical trial metrics offer insights, the purpose of incorporating any analytics in clinical trials is to drive demonstrable action and results²; without demonstrable action and results, analytics are futile. Proper risk and issues mitigation does not require only identifying analytics that can measure those risks/issues (KPIs/KRIs), but also incorporating business processes (actions) to ensure that risks are avoided during study operations (results).

To elaborate, let’s say that a study site is responsible for handling a costly investigational biologic, which needs to be compounded and prepared before it is administered to the patient. To add risk to the scenario, the biologic degrades after two hours from preparation and loses its potency, directly impacting study efficacy endpoints. Developing a KPI/KRI to measure the time from preparation to administration is not a sufficient quality control measure to ensure compliance; simply measuring that there was a time deviation after the biologic was administered is not a proactive approach. The sponsor should further mitigate this risk upfront by incorporating risk mitigative business processes and check control systems at the site, so that the site’s compounding center and research coordinators can (a) ensure that the patient is arriving for the study visit, so that no investigational product (IP) is prepared and wasted on a no-show, and (b) communicate and coordinate IP administration to avoid giving the patient degraded IP, which could adversely impact effectiveness outcomes.

Solely relying on KPIs/KRIs and expecting study sites to not miss critical study risks is not proactive. Incorporating business processes to prevent risks/issues from happening in combination with KPIs/KRIs is an effective way to oversee compliance, manage study performance and mitigate study risks.

What about unknown unknowns?

The benefit of standardized metrics is that they enable study teams to measure expected and commonly recurring issues and risks in study operations on a scalable level. Nonetheless, many study risks are not uncovered until trials start operating and study teams face new and unknown challenges.

Susan Callery D’Amico, Abbvie’s VP of R&D & Quality Assurance, on TransCelerate’s Quality Management Framework⁴, suggested that defining and categorizing ‘issues that matter’ upfront can assist with selecting the most appropriate quality risk thresholds for a study³. Callery-D’Amico goes further by advising that it is important to collect all ‘other issues’ in separate repositories, as data from those issues in aggregate may later become an issue that matters.

To demonstrate, defining screen failure categories upfront and collecting screen failure data during trial execution can help with uncovering impediments that are impacting enrollment (issues that matter). While these predefined categories are important, developing a separate category for all other screen failures (other issues) can uncover trends that were initially unknown, and subsequently, inform better inclusion/exclusion criteria optimization during protocol amendments⁵. Further, the use of analytical methods to research and uncover trends in ‘other issues’ can create a new KPI/KRI that can be added to an existing analytical portfolio, offering broader oversight on continuously changing study risks.

Summary

The standardization of clinical trial metrics is offering numerous benefits towards overseeing clinical trials, optimizing clinical operations, and mitigating study risks. Comprehensive and standardized oversight can assist with decision-making, nonetheless, there are always going to be situations where customized metrics will be needed in order to address study-specific risks, uncover unknown issues, and demonstrate actionable results.

References