Testing the RBM Waters—A Global Midsize Sponsor Weighs the Benefits of RBM Implementation

Risk-based approaches to monitoring (RBM) have become widely accepted in the clinical trials industry. Key milestones include the amendment to ICH-E6(R2) guidelines on November 9, 2016¹ and the adoption of the same by regulatory agencies in various regions-Europe (European Medicines Agency) and Canada (Health Canada)². Other countries, including Japan and Australia, are expected to follow suit. The FDA has also indicated its encouragement of the new model through the guidance document in March 2018.3

In general, large organizations (sponsors and CROs) have begun implementing risk-based approaches to monitoring. It is fathomable that since large organizations command a greater share of drug development and outsourcing of clinical studies, the necessity to initiate and adopt new strategies and innovative models is a strategic imperative for them. 

In contrast, due to limited financial and operational resources, smaller organizations face multiple challenges when adapting to rapid industry changes. Process modification, re-engineering SOPs, and operational change management can be daunting. Moreover, the development of new technologies requires upfront capital expenditure, which is risky when the model is evolving. All of these factors contribute to the slower adoption of RBM in smaller organizations. 

However, the need to comply with regulatory requirements leaves little room for organizations to delay implementation of ICH E6(R2). Hence, organizational readiness is important when such a time comes into effect. This case study seeks to share the experience of a sponsor organization that decided to utilize RBM in a limited, but effective manner.

A global midsize biopharma sponsor developed a promising anti-infective drug into a Phase II trial. In 2016-2017, the sponsor conducted a global study, spanning 40 sites across seven countries and several continents, including North America, Europe and Africa.The sponsor team believed in risk-based monitoring and the potential benefits of that approach, but was not in a position to fully implement RBM for a variety of reasons, including a reliance on the processes and procedures established by the CRO. Since RBM was in its nascent stages at that time, and the industry had not fully understood the impact, the sponsor utilized RBM in a limited manner, albeit with clear expectations and expected benefits.

Given the importance of the clinical study and its potential impact on business strategy and growth, the sponsor’s primary focus was to ensure that the study conformed to the highest standards. Hence, monitoring the overall study from a risk and quality perspective was the key objective of adopting RBM.

Risk Planning

Given the limited scope of the exercise, a simple yet objective risk-planning process was performed by the clinical operations and medical monitor from the sponsor team in consultation with a subject matter expert (SME).

For the global midsize sponsor, risks that could significantly affect the outcome of the study conduct were considered. Key performance indicators (KPIs) and key risk indicators (KRIs) were developed to objectively monitor the data in an efficient manner on a regular basis. The RBM SME facilitated and advised the risk-identification process, and developed the risk-assessment and monitoring framework. 

Acuity Analytics - an integrated data analytics and RBM enablement platform-was utilized for developing data visualization and custom analytics for monitoring the KPIs, KRIs and specific critical data. The custom dashboards and analytics were designed in alignment with the sponsor’s specific needs. Data sources included an electronic data capture (EDC) system and the CRO’s clinical trial management system (CTMS). These were integrated with Acuity Analytics platform. Data from the EDC and CTMS were refreshed every 24 hours and once a week, respectively.

Study start-up phase

During the start-up phase of the study, the team selected three key performance indicators to monitor the progress of study setup at the sites. These included:

• Site selection to site initiation visits at different stages

• Essential site document finalization and approval

• Site contract finalization and execution

Validated Clinical Operations dashboards from the platform’s data visualization library were used to understand the delay points at various process operational touch points with sites during study start-up. Delays (at the site, CRO or sponsor level) were monitored so that proactive and timely intervention could be taken. The sponsor’s operations team and central monitor performed periodic reviews. Outputs were shared with different stakeholders, and timely action was taken on a regular basis.

Figures 1-3 depict a few of the operational dashboards utilized.

Fig 1: Regulatory document turnaround

Fig 2: Essential documents readiness for Investigational Product release

Fig 3: Potential ineligibility

Study execution phase

From the risk-planning process, six KRIs were developed and utilized during the active phase of the study. Validated dashboards from the Acuity Clinical Insights library, along with analytics related to efficacy endpoints were aligned to the sponsor’s needs. The following are the KRIs and critical data points that were developed:

·      Adverse event rate

·      Missing clinically acquired bacterial pneumonia (CABP) data points

·      Missing eligibility (specific) data

·      Potentially ineligible subjects utilizing surrogate data points

·      Pharmacokinetic (PK) sampling deviations

·      Major protocol deviations

Figures 4-6 depict a few of the dashboards utilized for the central monitor’s review.

Fig 4: AE rate

Fig 5: PK sampling deviations

Fig 6: Risk-scoring depiction

A central monitoring plan was created during the initial phase of the study. The central monitor reviewed the KRI and critical data dashboards at frequent intervals (every four weeks) as per the central monitoring plan. The outputs were shared with the sponsor on a regular and timely manner. For certain types of complex data review that are prone to human error, custom algorithms were utilized to automate initial identification of issues. This was specifically applied for detecting potential eligibility violations and non-congruence of CABP symptoms in subjects as per the study protocol. In the event of specific issues, trends or patterns, more frequent reviews were performed. The outcome of the review was also recorded in a data-driven approach as per the criteria below.

Presence or absence of critical data

• Potential eligibility violations

• Missing CABP symptoms value

• Missing eligibility data

Extent of deviation from mean

• Adverse events (AE) per subject

• Pharmacokinetic (PK) sampling compliance

• Protocol deviation

Subsequently, the recorded outcomes from central monitoring were fed into a site-risk assessment model designed for Acuity’s Decision Factory. The model included the following risk thresholds and risk categorization illustrated in Table 1:

Additionally, the number of subjects enrolled at the site was factored into the model. The assumption is that more subjects at a site are directly proportional to increased operational and scientific workload for the site study teams and therefore, higher error rates. If such a scenario occurs, the impact on the safety of subjects at the site and scientific outcomes can jeopardize the overall study outcome. Needless to mention, additional operational and financial resources are required to bolster quality control and monitoring efforts. Hence, it was deemed fit to factor this into the risk-assessment model.

Results

Since this was a short study, the RBM approach to monitoring was conducted over a period of 11 months. The start-up phase of the study was monitored for four months, and the active recruitment and follow-up phases spanned more than six months. The study was successfully executed as per the originally planned timeframe.

The three KPIs as mentioned in the methodology section were reviewed using relevant data visualizations. Figures 1 and 2 illustrate some of the data visualizations that were utilized.

Continuous monitoring of study start-up delays were performed by the sponsor’s operations team that enabled them to review process delays at the site and enhance governance of the CRO that managed the conduct of activities at the sites.

The six KPIs as mentioned in the methodology section were reviewed using relevant data visualizations. Figures 3 to 5 illustrate some of the data visualizations that were used for review by central monitors.

After the review by central monitors against the central monitoring plan, the review outcomes were recorded in the site-risk assessment framework and factored in the subject load at the site as depicted in Figure 6. The time-to-date, site-risk assessment was performed over the duration of central monitoring by comparing from the baseline to understand progress over a period. The team used a normal data distribution test to segregate sites that had significant increase and decrease of risks from baseline. Key outcomes of the analysis were:

• Initially, 27% of sites (eight sites) were identified as high, 45% (15 sites) as medium and 28% (nine sites) with low-risk propensity. At the end of implementation, there was an overall reduction of risk propensity at sites. About 63% of the sites reduced in risk propensity from baseline.

• Two sites (6.25%)-categorized as high risk at baseline-continued to remain high risk until the last central monitor review cycle. 

• Five sites (15.65%) from the high-risk category were downgraded to medium or low risk.

• The KRIs that contributed to significant change from baseline were:

o  Missing eligibility (21%)

o  Missing CABP values (21%)

• Outputs from risk assessment were utilized for assigning resources for co-monitoring visits and quality monitoring visits. The sites were selected on a more objective basis than on purely subject recruitment reasons or issue escalations.

Conclusion

This study had limited scope to implement RBM. However, this exercise proved to be a precursor for future clinical studies, and the observed significant benefits. The operational benefits included:

• Data visualization and custom analytics helped in central data monitoring.

• Data visualizations afforded an intuitive and significant efficiency in data review. The data review process time frame was reduced by about 45% from the first data review process.

• The site-risk-assessment approach was objective and driven by data. The congruence of the model to ground realities observed by site monitors was validated by seeking inputs from the sponsor’s clinical operations team.

• The approach gave impetus to formally adopt and conduct risk-based approaches to monitoring in future studies in a scalable manner.

Stakeholders from this exercise also learned certain lessons. To a great extent, this could not have been incorporated as the expectation was to keep it light from an execution perspective. 

• Laboratory data transfers into EDC need to be more frequent to enable better and close to real-time central data review of subject data.

• Ensuring a formalized and more comprehensive risk-planning approach so that other cross-functional inputs, as well as risks, can be captured.

• Though feedback of central monitoring was utilized for downstream action, a formalized risk governance process would aid in changing and re-prioritizing KRIs during the course of the study execution. This would also entail iterating the risk plan during the course of the study. 

Understandably, operational leaders often enquire about the business or financial benefits, which are as follows: 

• The total cost of implementation was about 0.6% of the total clinical budget. This incremental expense in the overall scheme of study conduct expenditure and the stakes involved in the molecule’s clinical development was well justified. There were no critical findings from Quality Assurance audits. No significant findings that affected safety, scientific or quality outcomes of the study were observed. 

• The success of this exercise gave impetus to the operational team and leadership to pursue risk-based approaches to monitoring in future studies. It is now an integral strategy of clinical operations conduct of studies through collaborating CROs.

In summary, this is a good case in point of how small and midsize organizations can implement risk-based approaches to monitoring. The baby steps taken in a pilot study, along with an expert’s help, represent a plausible way to get hands-on understanding of the nuances of execution as well as foresee challenges when RBM requires implementation in a scalable manner on more studies. It helps in formulating the right interventions for adoption and change management between different stakeholders earlier on when organizational processes and SOPs need modification. Utilization of technology will be key and could level the playing field between large and small organizations. Research organizations can overcome apprehensions about investing in comprehensive enablement technology by procuring easy-to-use, smart and customizable innovative technologies on a study-to-study basis. This can be a quick-and-easy starting point before pursuing a strategic roadmap.

References

1. Integrated Addendum To ICH E6(R1): Guideline For Good Clinical Practice E6(R2) Dated 9 November 2016.
2. Guidance Document-Good Clinical Practice: Integrated Addendum To E6(R1) ICH Topic E6(R2).
3. E6(R2) Good Clinical Practice: Integrated Addendum to ICH E6(R1) Guidance for Industry.  

Abby Abraham, Vice President of Data Analytics and Risk-Based Monitoring at OmniComm Systems, Inc. and Ranjeet Gutte is the General Manager, Global Clinical Development for Wockhardt Ltd.