Protocol Amendments: a Costly Solution

Protocol amendments are a fact of life in clinical trials. They occur frequently. Sponsors, CROs, and investigative sites seem to simply put up with amendments and their associated study delays and incremental costs.

Kenneth A. Getz

True, clinical research professionals perceive protocol amendments as a major and largely unavoidable problem, but it is more appropriate to view amendments as solutions to underlying problems that include cycle time pressures, difficulty recruiting study volunteers, and rising protocol design complexity. In a 2008 study that I conducted with my colleagues at the Tufts Center for the Study of Drug Development (Tufts CSDD), for example, we found that less complex protocols—those containing fewer procedures and eligibility criteria—averaged two amendments; more complex protocols averaged 3.2 amendments.

But with the exception of this study finding, little data can be found in the literature on protocol amendments, why they occur and their consequences. In an effort to benchmark the incidence, causes, and impact of protocol amendments, Tufts CSDD convened a group of 17 mid-sized and large pharmaceutical and biotechnology companies in 2010. Participating companies contributed data on their protocol amendments and their implementation. The results of this study were released this month. The study offers insight into the cost and time required to implement protocol amendments, ways to better anticipate and plan for their impact, and the opportunity to reduce, and even prevent the need to implement certain protocol amendments in the future.

In all, sponsor companies provided data on a representative range of therapeutic areas from 3,410 protocols approved between January 2006 and December 2008. Protocols approved within the most recent 12 months were excluded from the study because these did not have enough time to accumulate amendments. Amendments were defined as any change to a protocol requiring internal approval followed by approval from the IRB/ERB or regulatory authority. Detailed data on 3,596 amendments containing 19,345 total protocol modifications was analyzed.

Companies participating in this study detailed specific changes made per amendment yielding a total of 6,855 changes classified. Companies also provided the top causes for amendments and rated each amendment in terms of whether it was "Completely Avoidable," "Somewhat Avoidable," "Somewhat Unavoidable," or "Completely Unavoidable."

Incidence, occurrence, and cause

Nearly 60% of the 3,410 protocols required one or more amendments. Completed protocols across all phases had an average of 2.3 amendments, though later-stage Phase II and III protocols averaged 2.7 and 3.5 amendments respectively. Each amendment required an average of 6.9 changes to the protocol. Therapeutic areas that had the highest incidence of amendments and changes per amendment include cardiovascular and GI protocols.

Of the 6,855 changes categorized, 16% were modifications made to the description and eligibility criteria of the patient population under investigation; 12% were adjustments made in the number and types of safety assessment procedures; and 10% were edits and revisions made to the general information contained in the protocol (e.g., protocol title and study staff contact information).

Nearly 40% of all amendments occurred before the first study volunteer received first dose in the clinical trial. This was most pronounced in phase I studies where 52% of amendments occurred prior to beginning patient enrollment. In Phase II, III, and IIIb/IV studies, 37%, 30%, and 38% of amendments occurred before first patient first dose, respectively.

The most common cause of amendments was the availability of new safety information (19.5%), followed by requests from regulatory agencies to amend the study (18.6%), and changes in the study strategy (18.4%). Protocol design flaws and difficulties recruiting study volunteers were also top-cited causes at 11.3% and 9% of categorized amendments, respectively.

Two-thirds (66%) of amendments had causes that sponsor companies considered unavoidable, including amendments that were the result of new safety information, new regulatory requests, changes in the standard of care, or study objectives.

But of all amendments implemented, 34% were considered partially or completely avoidable. To repeat that statistic: One-third of all amendments were avoidable. Undetected design flaws, inconsistencies, or errors in the protocol, and difficulties recruiting study volunteers were rated as amendment causes that sponsors could have better anticipated and avoided. This finding combined with the high percentage of amendments occurring before first patient first dose, strongly suggest the need for, and impact of, better upfront protocol planning and review.

Time and economic costs

Most companies were unable to provide amendment implementation cycle time and cost data in this study. As such, these results need to be viewed with caution, as an initial attempt to quantify operating impact. Based on data from 59 amendments, data gathered on the median total cycle time from when the protocol problem was identified to when the amendment was fully implemented (i.e., first patient resubmitted under the revised protocol) was more than two months (65 days). On average, it cost $453,932 to implement a single protocol amendment. This figure is undercounting the full economic impact as it does not include the cost of internal FTE time dedicated to implementing each amendment, costs or fees associated with protocol language translation, and costs associated with resubmission to the local authority. The largest areas of amendment-associated costs were increases in investigative site fees (58% of total costs) and contract change orders with CROs and other third party providers (24%).

The total cost for sponsors to implement "avoidable" protocol amendments each year is approximately $2 billion. This is based on a crude, back-of-the-envelope estimate using the incidence of amendments (60% of protocols with an average of 2.3 amendments for each protocol), the cost to implement them ($1.04 million on average), and the proportion that is avoidable (33%). Given 8,386 active protocols involving FDA-regulated products in 2009, industry paid upwards of $1.7 billion in direct costs to implement avoidable amendments. This estimate does not include the billions of dollars realized annually in aggregate time-savings and earlier commercialization due to the elimination of each avoidable amendment.

Necessary nuisance

The results of this study do not indicate or suggest that sponsors should discontinue making protocol amendments. Ideally, amendments enhance protocols and ensure more scientifically successful clinical trials. Many amendments are necessary to optimize study results and ensure patient safety and their ethical treatment.

However, sponsors must also recognize and balance the operating needs of their trials. Most companies choose to implement protocol amendments as a conservative practice resulting in an unanticipated run-up in study budgets and cycle times. Amendments—as this study shows—substantially and adversely impact clinical trial feasibility and operating performance.

There are many reasons why sponsors have historically dismissed protocol amendments as a necessary nuisance. Clinical research scientists and project managers involved in large scale studies typically perceive the marginal cost of amending a protocol and collecting additional data trivial in comparison with the overall study budget. Many believe that protocol modifications and the additional data gathered may prove useful in addressing future regulatory agency inquiries and will assist in deriving new insights for commercial application.

But the high cost, cycle time delays, and increased operating complexity of implementing protocol amendments identified in this Tufts CSDD study suggests that the perception of trivial costs may be grossly inaccurate. Clinical research scientists and operating managers need to be aware of, and to weigh, the potential costs associated with protocol amendments when making the decision to amend or not to amend a protocol.

This study found that the most common causes of protocol amendments are regulatory agency requests, new safety or dose related information about the study drug, new standards of care, competitive pressures, protocol design inconsistencies and flaws, and patient recruitment difficulties. These causes reflect a wide range of choices that clinical study teams must consider to manage performance. Some of these causes also reflect judgments made, and risks taken, by clinical research management to meet ever-tighter deadlines and budget requirements. Faced with time pressures, project teams often must move forward at risk while pending announcements of new standards of care and regulatory decisions loom.

Intensely ambitious and demanding protocols are at the heart of the problem. Given the demonstrated positive correlation between protocol complexity and the incidence of changes made to the protocol, an important first step in reducing amendments rests with simplifying clinical research study designs.

At a minimum, these study results offer insights and refinements into project planning and budgeting expectations. The results of this recent study also suggest that sponsor company efforts to reduce and prevent certain amendments through better protocol designs and more informed decision-making would improve study feasibility and efficiency and yield substantial cycle time and budget improvements.

Kenneth A. Getz MBA, is a Senior Research Fellow at the Tufts CSDD and Chairman of CISCRP, both in Boston, MA, e-mail: kenneth.getz@tufts.edu