Conquering RBM: A Story on Medtronic's Risk Based Management Methodology

I have been tracking and writing about risk-based monitoring (RBM) since the inception of FDA’s initial draft guidance in 2011, and I am finding that biopharmaceutical sponsors are still struggling to implement RBM. After speaking to Jonathan Helfgott, Associate Director for Risk Science at the FDA, I asked him why the FDA kept their RBM guidance open to interpretation; Helfgott suggested that every company is different, and they should feel free to exercise whichever methodologies they feel are appropriate for their clinical trials.

The industry has tried to establish some consistency by standardizing RBM methodologies. Amgen, for example, has leveraged its FSP model to execute RBM, and Bristol-Myers Squibb (BMS) appears to be getting closer towards establishing an infrastructure that supports RBM, moreover, Transcelerate continues to release elaborate and complex RBM guidance documents. Personally, I have not seen actionable advances in RBM until I met Peggy Fay, Director of Global Clinical Monitoring, Medtronic Clinical Operations.

About a year ago I met Peggy at a New York ACRP symposium and there were several topics presented, one of which included Risk-Based Monitoring. Peggy was faced with a monstrous challenge: she was overseeing a multi-centered medical device clinical trial with 1500 patients and due to complex protocol design, 2970 data points needed to be collected, monitored and verified. Given Medtronic’s monitoring methodologies at that time, Peggy estimated it would cost more than $21 million to monitor that data. “It was not financially viable,” said Peggy, as she indicated that number of monitors needed to cover 45 sites and to review 218 CRF pages would blow through the study budget. “We had no choice but to find a more efficient way to better manage and monitor the data, to ensure data integrity and subject safety” added Peggy.

Facing the Organization

Many biopharmaceutical enterprises are still coping with executing successful RBM pilots because of organizational challenges. BMS, for example, had to implement a change management program that focused on education and communication of a new set of roles and responsibilities in order to enable RBM.

Peggy took a different approach; why not make RBM mold to existing roles and responsibilities? Why not build a system and process that is intuitive and easy to use in nature, given current study team skillsets and operational roles? “When I ask people about RBM, they think Risk-Based Monitoring. We think about Risk-Based Management,” said Peggy. Peggy stressed that risk-based management is not strictly about monitoring, but rather, quality risk-based management across a clinical trial. Medtronic’s risk-based management initiatives were fueled by several internal and external drivers; internal drivers for change included questionable data integrity, site management/performance issues, and unproductive resolution findings due to communication concerns between the sponsor and monitors. External drivers included minimal changes in protocol violation and nonconformance rates, and regulatory pressures for improved quality risk management.

Peggy rounded up a variety of clinical operations groups and started having meetings on changing internal processes to formulate and support the risk-based management infrastructure. “It can be difficult to put cross functional teams together, getting them to focus on a common goal at the same time you are trying to change a ‘mindset’. But, we were forced to collaborate, to be creative as collectively we were all faced with an unsustainable situation, we had no choice,” said Peggy. 

Medtronic’s ‘Plan, Do, Check, Act’ Process

Peggy believes that risk-based management is a systematic approach towards identifying and mitigating risks. This process involves adaptive monitoring, defining critical versus noncritical risks, meeting study and regulatory needs, and developing tools to incorporate risk management strategies into clinical trial design. Medtronic created a risk management process to address these issues, namely, Plan, Do, Check, Act, which is supported by FDA Guidance.

Jean Mulinde, Senior advisor of CDER at the FDA indicated in a recent DIA workshop that the FDA considers protocol design as the blueprint for quality and demonstrated how poorly designed protocols resulted in an increase in FDA audits and adverse findings. Medtronic Clinical Operations (MCO) has embraced this philosophy, as in the Plan portion of the process, they have incorporated quality by design methodologies on critical study endpoints during protocol design.  During this process, a cross functional team conducts a risk assessment, impact analysis and risk mitigation strategy; defining critical and noncritical risks associated with the study, patient population and/or study sites. Assigning weighted values to risk indicators the team sets monitoring intensity and assigns centralized/on-site monitoring activities. Subsequently, the study team and functional service providers generate risk-based monitoring, data management, safety and SAP plans that each identify corresponding communication, escalation and action plans if Key Risk Indicator (KRI) signals are triggered.

The MCO monitoring team uses a gradated KRI model, as illustrated in Figure 1.

The Do and Check portions of the process focus on study execution. In the Do portion, Medtronic targets measuring critical data (i.e., predefined data points that are relevant to primary/secondary endpoints and are considered high risk) from two perspectives: (1) at each site, and (2) aggregated sites across the trial. Centralized monitors focus on data quality, site responsiveness, subject retention, and protocol compliance, whereas on-site monitors focus on targeted source document verification (critical data), and random statistical sampling. 

In the Check portion, Medtronic analyzes a variety of risks, such as intrinsic risks (i.e., safety profiles, site experience, study populations, AE tracking), design risks (i.e. protocol simplification, number of protocol amendments, and study endpoint approach), and study operational risks (i.e., site staff turnover, site workload, enrollment rates, AE reporting levels). Figure 2 demonstrates Key Performance Indicators (KPI) for specific KRIs.

Figure 2: Site Risk Indicators

Figure 2 shows that this particular study site was experiencing a resource constraint; the study site was enrolling too many patients to handle the necessary workload that is associated with data management discrepancy resolution, and AE reporting. “We not only focused on under-enrolling sites, but, we were concerned with over-enrolling sites; were they prepared to take on that many patients? How deep were the bags under the coordinator’s eyes? This tool allowed us to better oversee site activities and help sites improve performance through early intervention,” mentioned Peggy.

In the Act portion of the process, the MCO study team and monitoring group are able to initiate a proactive approach via CAPAs to mitigate issues before they build, which minimizes costs, and improves data integrity/transparency, especially during FDA audits. Further, the MCO monitors are able to continually evaluate risk-based risk and performance indicators, driving escalation plans in response to variances or issues of non-compliance thus ensuring a self-innovating cycle.

No Issues with FDA Audits

I have observed that many clinical operations personnel have a tendency to exhibit a level of hesitancy when it comes to validating RBM systems with FDA audits. On the contrary to status quo beliefs, well-designed RBM systems and processes protect biopharmaceutical enterprises from adverse findings during FDA audits, and the Medtronic case study validates the fallacy. To confirm the value of the RBM approach, the MCO operations team awaited results from multiple site audits; no significant concerns or observations noted during these audits. Peggy confirmed the risk-based management model allowed the study team to generate a well-designed protocol that focused only on critical data; together with cross functional team the collaborative process improved transparency, data quality and integrity, and supported a proactive response to event signals compared to traditional monitoring and operational models.

Be Flexible and Practical

In summary, while the industry continues to run around in circles trying to figure the best fit model for RBM, the key towards implementing successful RBM strategies include executing a practical process that is flexible towards an organizations specific infrastructure. The Medtronic Clinical Operations (MCO) approach is a great example of how a medical device company was able to successfully (1) change clinical operations cultures to adopt risk-based management philosophies and processes in study and clinical operational design (2) develop simple and practical analytical tools to not only quantify and measure risk, but also improve data transparency (3) mitigate risks through CAPAs and proactive targeted monitoring, and (4) continually improve risk-based management systems. 

Despite the challenges associated with internal bureaucratic hindrances and restrictive cultures, it pays to take the leap of risk in order to benefit from risk-based management systems, and Peggy has successfully adopted innovative methodologies in RBM that many biopharmaceutical companies eagerly vie for. I hope to see operational advancements in RBM methodologies at

CBI’s Risk-Based Monitoring in Clinical Studies Conference from November 6-7 in Philadelphia