Quantifying Protocol Deviation Experience by Clinical Phase

It is widely recognized that the clinical research enterprise embraced numerous changes and adaptations during the COVID-19 pandemic to maintain patient safety, accelerate the development pathway, and ensure clinical trial continuity. Topping the list of changes is the unprecedented number of protocol deviations that have been implemented since the global lockdown.

Sadly, we do not have empirical data at this time characterizing the expected sharp increase in the average number of deviations per protocol. However, my team at the Tufts Center for the Study of Drug Development (Tufts CSDD) has recently completed a study capturing pre-pandemic protocol deviation metrics from which future comparisons can be made.

Definitions and Impact

Typically, protocol deviations are executed by study staff while the clinical trial is underway in an effort to accommodate study volunteers encountering difficulties complying with the schedule of visits or adhering to study medication administration requirements. Investigative site personnel may also deviate from the protocol when facing difficulties finding eligible patients, and following and performing clinical and administrative procedures dictated by
the protocol.

Most deviations have a minimal impact on protocol integrity and execution. A subset of all deviations, however, are more significant and may harm the quality of the research, may put the rights and safety of study volunteers at risk, and may detract from the completeness, accuracy, and reliability of the study data. There are a number of useful resources to help organizations identify, classify, and manage protocol deviations including those by the Drug Information Association’s (DIA) Good Clinical Practice and Quality Assurance community and by TransCelerate BioPharma.

Finally, its important to note that protocol deviations are the top cause of FDA inspection warning letters. Approximately one-third (30%) of all warning letters are due to the failure to follow the investigational plan. Site failure to adequately maintain case report records is the second most common reason, accounting for 17% of all warning letters.

A baseline assessment

To our knowledge, little-to-no data exists quantifying and benchmarking protocol deviation experience overall and by clinical phase. For this reason, Tufts CSDD engaged a working group of 20 major and mid-sized pharmaceutical companies and contract research organizations (CROs) to gather hard data on deviations associated with traditional clinical trial protocols (i.e., master protocols and adaptive designs were not included). Our study sample included protocols that had received final approval between January 2013 and December 2018 and had a primary completion date or database lock date before December 31, 2019. The data-collection process deployed in this study followed an approach that Tufts CSDD has been using to evaluate trends in, and the impact of, protocol design practice since 2008.

Data on a total of 187 protocols targeting a range of disease conditions was analyzed. One-quarter of the total were Phase I protocols, 39% Phase II, and 36% were Phase III protocols.

The table on the facing page provides high-level baseline measures of protocol deviation experience by clinical phase. Phase I protocols have the lowest mean number of deviations and involve half the proportion of patients than those observed in Phase II and Phase III protocols. Each Phase III protocol has a mean number of 119 total deviations, involving approximately one-third of all patients participating in that protocol. The coefficients of variation around the mean number are very high—between 1.7 and 2.0—indicating that the average deviations per protocol are widely dispersed and difficult to predict.

Wide variation in protocol deviation experience was also observed by therapeutic area. Phase II and III oncology protocols average almost 20% more total deviations than the average for non-oncology protocols, at 108.8 and 91.9, respectively. Protocol deviations in combined Phase II and III oncology protocols involve 47% of the total study volunteers, nearly double the proportion observed in non-oncology protocols. Rare disease indications average a lower relative number of total deviations (78.1) among a smaller proportion of study volunteers (27.7%) compared to that of non-rare disease indications.

Tufts CSDD also found that the incidence of protocol deviations was significantly, positively correlated with all clinical trial milestone durations, with the exception of study start-up and study close-out. The duration to complete all first-patient visits (first patient first visit to last patient first visit), the overall treatment duration (first patient first visit to last patient last visit), and the total clinical trial duration from protocol approval to database lock showed weak but significant positive correlations with the number of protocol deviations per protocol.

We were not surprised to find that no association existed between the incidence of protocol deviations and study start-up and close-out cycle times. Protocol implementation burden is generally low during the study initiation process. This burden, however, for both study staff and study volunteers, becomes more pronounced as these parties must comply with protocol implementation requirements over time.

A relationship between select protocol design complexity variables and the incidence of protocol deviations was observed. The number of endpoints and the number of procedures per visit were modestly positively associated with, and predictive of, the incidence of deviations per protocol. Adjustments to these select design variables may play a role in preventing protocol deviations and in minimizing their risk to the study and data integrity.

Two operational variables were also positively associated with protocol deviations. The number of countries showed a modest relationship. A positive relationship was also demonstrated between the number of investigative sites and the number of protocol deviations, suggesting that decisions and behaviors that diverge from the investigational plan may be a function of variation in investigative site personnel training, site infrastructure, operating environment, and culture.

Building on this research

Tufts CSDD plans to build on this body of research. We will be looking at the root causes of protocol deviations and how to address them. Future research will also look at deviations with more granularity to understand the incidence and impact of minor and major deviations. Whereas some deviations may improve participation convenience and clinical trial executional feasibility, other deviations may pose substantially greater risk to study volunteer rights and safety and to the quality and integrity of the clinical trial data. We also hope to understand the relationships between types of deviations and patient characteristics—including demographic subgroup and geographic location—and investigative site characteristics.

The results of this initial study provide useful measures that sponsors and CROs can use to benchmark their own protocol deviation experience and identify factors most associated with protocol deviations.