MCC Metric of the Month Blog: Protocol Amendments

MCC has 100+ standardized metrics spanning timeliness, cycle time, quality, efficiency, cost and risk, as well as a new Metrics Search Engine to help you identify the most appropriate metrics for your situation. This month, let’s look at a quality metric that’s critical to on-time, on-budget performance: protocol amendments.

Why this metric is important:  In a May 2011 article in Applied Clinical Trials, Ken Getz discussed the results of a Tufts study on amendment frequency and costs. The results are sobering: more than 3 amendments per protocol in Phase III studies comprising roughly 25 changes and costing over $1.5MM. Industry-wide, Tufts estimates that amendments cost companies roughly $6B/year, and that at least $2B of that sum is avoidable. If we can track our amendments, learn from them and avoid them in the future, we stand to reap huge savings in time, cost and frustration.

Definition:  The MCC protocol amendment metric is calculated as the number of protocol amendments after protocol approval, excluding agency-required changes, standard of care changes, and emerging safety data. These excluded amendments are often referred to as “unavoidable” amendments, while the included amendments are often referred to as “avoidable” amendments. The overall goal is to reduce the number of avoidable amendments; those things that could have been anticipated through a robust protocol quality, feasibility and enrollment validation process. The metric can be calculated at the study, indication, therapeutic area and company level, and should be sliced by Phase (later phases tend to have more amendments).

How to calculate this metric:  The formula is simply the [total number of amendments] minus the [number of excluded (unavoidable) amendments]. If an amendment has both included and excluded changes, a fractional amendment can be included.

Calculating fractional amendments requires some simple analysis, since most amendments incorporate multiple changes, only some of which should be excluded. The example below provides a technique for doing this.

This metric should be calculated once for each study: when the study is complete.

<1 avoidable amendment should be the target value for this metric.

Example:  In the table below, you can see that a particular protocol has had 4 amendments. The first amendment includes 7 changes but column B shows that none of them has any excluded causes (i.e., all were avoidable) and thus is counted as a full amendment in column C. The second amendment has 3 changes, but all are due to excluded causes (column B), so column C shows a value of 0. Amendments 3 & 4 have a mixture of included and excluded changes and are shown in column C as fractional amendments. In total, then, our amendment metric is 2.5, even though there were a total of 4 amendments.

What you need in order to measure this:  You need the following three things for each study:

• the number of amendments

• the number of changes in each amendment

• a tally of which changes in each amendment are included and which are excluded

What makes performance on this metric hard to achieve:  Study teams are typically in a rush to complete the protocol and begin execution. Amendments are considered a cost of doing business and are not perceived as important during the protocol development stage. Thus it is difficult to persuade teams to take the time to scrub and “optimize” their protocol before finalizing it.

Things that you can do to improve performance:  Once you are tracking this metric, you can track the predominant types of amendments, as well as the causes that can be avoided. You can then build in techniques for avoiding these causes and amendments in future studies.

A useful tool for helping to avoid amendments is the Protocol Quality Tool developed by MCC and available to members.

Companion metrics:  Other metrics that you should consider in tandem with this metric include: (1) the MCC Protocol Quality metric and its related scoring tool, (2) the MCC Protocol Deviations metric, and (3) the MCC Planned vs. Actual Screen Failure Ratio.

Dave Zuckerman, CEO, Metrics Champion Consortium, [email protected]
Linda Sullivan, COO, Metrics Champion Consortium, [email protected]