Simulation: A Critical Tool in Adaptive

In recent years there has been increased industry interest and utilization of adaptive clinical trials. Although the term "adaptive" covers a large range of study features and designs, much of the current excitement is around designs that enable treatment groups to be dropped during the trial to enable more doses to be investigated and/or to reduce time between development phases using seamless designs.

PhotoGraphy: Bruce Laurance, Getty Images Illustration: Paul A. Belci

Perhaps the most interesting of these applies to dose finding, where critical decisions regarding the dose to take forward into Phase III is sometimes made on limited information due to practical limitations regarding number of doses that can be investigated in Phase II studies. Taking a suboptimal dose into Phase III can result in having to repeat studies with different doses or can lead to incorrectly terminating development.

Of those drugs that finally reach the market, it is estimated that one in five is launched with a flawed dosage,¹ which can be expensive when discovered after pricing and reimbursement details have been agreed upon.

Implementation challenges

Despite the promise of adaptive designs, some sponsors consider their complexity a barrier to implementation. Challenges such as real-time collection of accumulated subject response and safety data and rapid implementation of randomization adaptations can be overcome by careful selection of integrated technologies.²

Another potential barrier relates to the planning of these designs. Specifically, determining the optimal characteristics of the study design can be a complex yet critical decision. Typical questions might include:

• How many interim analyses should be used?

• What sample size would be optimal at each interim analysis?

• Is a conventional design a suitable alternative?

• Should a Bayesian response-adaptive algorithm be used instead of preplanned interim analyses?

In addition to answering these questions, planning for an adaptive design creates new challenges, not least of which is the estimation of the quantity of drug supply required by the study. This is often a complex question for conventional designs, how much more so for designs that have the possibility of adjusting the proportion of subjects allocated to each treatment as the study progresses?

This article explores how simulation has become a vital tool in answering these important questions, enabling researchers to confidently plan and implement adaptive clinical trials.

Simulation

Simulation is a valuable tool used by many industries to understand and investigate the properties of complex systems. In clinical trials, simulation is commonly used to explore and optimize study design, for example, in selecting between different randomization methodologies to achieve the desired treatment balance. Monte Carlo simulation uses computer-generated random numbers to model real-life variability. This variability enables a true picture of the likely range of outcomes that might result, rather than a single average estimate.

In this article, we demonstrate via a case study how simulation can be used to explore the optimal statistical design for an adaptive trial and to estimate the supply requirements for this design. To achieve this we have combined two proprietary simulation tools. The first is a design simulator that enables the study and statistical analysis to be simulated, including planned interim analyses and decision rules, CytelSim.³ The second is a supply chain simulation tool that models the supply and use of medication packs in studies where the supply chain is controlled using Interactive Voice Response (IVR) systems, MedSim.⁴

Our novel approach has been to link the two simulation engines in such a way that the simulated profile of treatment allocations (including the dropping of treatment arms) generated by the adaptive design simulation can be used by the supply chain simulation in determining the study medication requirements.

As illustrated in the case study that follows, this combination of simulation tools enables researchers to confidently plan and implement these complex study designs, and further to allow the impact of supplies to be taken into account when determining the optimal study design.

Case study

We illustrate how simulation is a valuable tool in determining the appropriate study design and in supplies estimation by considering a simple case example.

Study design. The example is a dose finding study investigating six active dose levels—10 mg, 20 mg, 30 mg, 40 mg, 60 mg, and 80 mg—and placebo, with a total of 140 subjects to be enrolled. The first 70 subjects are randomized in an equal allocation ratio to all seven treatment groups, at which point recruitment is halted and an interim analysis is performed on the week eight efficacy response endpoint data collected from each patient.

At this analysis, dose levels that are unlikely to give clinically relevant differences from placebo are dropped, and the additional 70 subjects are randomized to those treatment arms that remain, again using an equal allocation ratio. The decision rule for dropping a dose level is based on the conditional power calculated from the observed responses of subjects up to the interim analysis. Conditional power estimates how likely it is that at the end of the study the dose level will be shown to be significantly different from placebo.

A further aspect of the decision rule is that if a dose is not dropped at the interim analysis, then no higher dose will be dropped. In this case, that dose level and any higher doses are continued into the remaining phase of the study. Where all dose levels are rejected, the study will be terminated at the interim analysis for futility.

At the final analysis the tests of significance for difference from placebo are carried out after adjusting for the interim analysis and for multiple testing by using methods developed by Posch and Bauer.^5,6 These methods guarantee that the type 1 error is controlled—at the one-sided 0.025 level in our case.

Design simulation. In simulating the statistical outcomes of the trial, we need to make some assumptions about the possible dose-response relationship and the distribution of the study endpoint. In this case, we assume the endpoint is normally distributed with a fixed standard deviation. The mean response at each dose level we assume to follow one of a number of possible scenarios. In practice, these assumptions may be strengthened by previous data or data from similar trials with other compounds, or may simply cover a broad range of possible outcomes where little prior information is available.

The scenarios we consider range from "no effect" (scenario 1), to an effect predominantly at the two highest doses (scenario 2), to a steady dose response across the complete dose range (scenario 3). We assume that the minimum clinically relevant improvement relative to placebo is a difference of 25 units in the mean response. Thus, the 60 mg dose is the lowest dose producing a clinically relevant difference in both scenarios 2 and 3 (see Figure 1).

Figure 1

Findings and conclusions. We performed 100 simulations of the study under each dose response scenario using a proprietary Monte Carlo simulation tool, CytelSim.³ The simulations show that a sample size of 140 with a threshold level of 0.2 for conditional power for dropping dose levels has good operating characteristics.

In scenario 1 (no effect), it was observed that in 51% of simulations we would stop the study for futility—that is, reject all doses including the 80 mg dose. For scenarios 2 and 3, where we assume a true effect to occur, the study would only be (erroneously) stopped for futility with a probability of 1% (see Table 1).

Predicting the Study Arms Dropped

The majority of simulations showed that in scenario 2, the 60 mg and 80 mg doses were continued following the interim analysis, which is the desired outcome should this scenario represent the true dose-response relationship. The same was observed for scenario 3, with the addition that in over 50% of simulations the top four doses would be continued, and in over 75% of simulations the top three would proceed following the interim analysis.

Again, this is a good outcome for this scenario, indicating that the number of patients recruited at the time of interim analysis and the choice of decision rule is likely to be appropriate and perform well for this study.

Supply chain details

Expanding the simulation to include supplies estimation and investigation of optimal packaging, the output of the statistical design simulation was linked to a Monte Carlo simulation model of clinical supply chain management, MedSim.⁴

In this study an IVR system was used to manage the supply inventories and treatment allocations. During the eight week treatment period, each subject received two dispensations of study medication, one at randomization (week zero) and a second at their week four clinic visit. Medication packs were numbered with unique pack numbers so that any pack could be allocated to any patient in the appropriate treatment group.

Two possible packaging configurations were considered. In the first, packaging scenario A, medication was packaged uniquely for each possible dose group, with a single pack allocated at each dispensation. In the second, more flexibility was introduced in that four packs were allocated at each visit, with each dose level being composed of a combination of placebo, 10 mg, and 20 mg packs—the rationale being that pack types for dropped arms at the interim analysis could still be dispensed for subjects randomized to the remaining arms postinterim.

In this study, this was an acceptable scenario as the dose is administered at site. In studies where subjects will be required to take their medication unsupervised, it might be determined that no more than two packs of medication be used concurrently to avoid adversely affecting compliance.

We assume there are 12 sites involved in the study, each recruiting on average 0.5 subjects per week over an approximate six month enrollment period. A single supply depot provides medication packs to each study site with a delivery time of between three and seven days. The IVR medication management process is assumed to follow a standard trigger-and-resupply algorithm incorporating a predictive algorithm for subject resupply visits. This is a standard IVR methodology that is described in more detail elsewhere.⁷

Simulation findings

Using the simulated profile of treatment allocations from the adaptive design simulations, we performed 100 simulations of the supply chain for each of the dose-response scenarios considered. In doing so, we ensured that the IVR supply management algorithms were optimized so that minimum stock levels of each pack type were maintained at sites while ensuring there were no occasions where the correct medication was not available for a new or returning subject.

We summarized output by considering the total number of packs and quantity of bulk ingredient required by the study, including packs left unused at each study site, and the number of deliveries of medication required over the course of the study.

Packaging scenario A: Single unique pack per treatment group dispensation. The maximum requirement of placebo packs will occur when only the 80 mg and placebo doses remain following the interim analysis, representing 10 placebo patients prior to the analysis and 35 recruited afterwards. With two packs dispensed per subject and additional packs required to maintain adequate site buffer stocks, simulations estimated a maximum requirement of 123 x 80 mg and placebo packs for the study (see Figure 2).

Figure 2

For each of the active dose groups, the maximum requirement under any scenario occurs when only that dose level and those above it are continued after the interim analysis. Hence, we estimate a declining profile in pack requirements as we descend dose levels, as postinterim requirements are diluted across more dose levels. The maximum requirements are similar under each dose-response scenario, as every outcome in terms of the doses that could go forward after the interim decision is possible.

Taking the worst case of all the scenarios we estimate a maximum medication pack requirement of 520 active packs and 123 placebo packs. Accounting for the dose level of each, this equates to a total of 23.7 g of active ingredient to make up the medication packs for this scenario. Additionally, the IVR system requested between 125 and 136 shipments of medication to maintain site supply inventories.

Packaging scenario B: Combination of four packs per treatment group dispensation. With this scenario we anticipate lower total medication requirements as we have introduced flexibility in the way packs can be utilized, and hence less wastage. For example, 20 mg packs can be used to make up any dose from 20 mg upwards. This expectation bears true in the simulation findings that indicate that in the worst case a maximum of 805 active packs and 781 placebo packs would be required to complete the study (see Figure 3).

Figure 3

Expressing this as a quantity of active ingredient, this represents 14.9 g active ingredient, a savings of over 37% compared to scenario A. In addition, the increased flexibility in pack usage led to a reduced requirement to resupply the sites. In this case, the IVR system required between 75 and 77 shipments to maintain site inventories.

Conclusions

The case study has illustrated the value of simulation in assessing design adequacy when planning an adaptive trial, and in estimating the supplies required by these complex studies. Together, this provides study teams with the ability to assess both the statistical properties and the impact on the supply requirements when evaluating and deciding between possible study design options.

Although it is generally recognized that there will be increased supply requirements associated with many adaptive trial designs, there may be options available to reduce the required overage, such as different packaging configurations (as in the case study) and producing supplies using multiple packaging campaigns throughout the study.

Flexible use of supplies across dose levels can return large savings in the quantity of medication required to operate an adaptive clinical trial. This can enable researchers to investigate more dose levels during dose-finding studies without increasing the supply requirements beyond practical or affordable limits.

The example we presented is deliberately simplistic, and there are many other areas we may want to explore through simulation before agreeing on the exact design properties and associated supply requirements. These include:

• Comparing the power of different study designs and the sample sizes required

• Exploring the effect of more than one interim analysis

• Considering different decision making criteria at each interim analysis

• Applying a Bayesian response-adaptive algorithm for continuous reassessment of the treatment allocation ratios as opposed to utilizing Frequentist methodology at fixed interim analysis

• Considering enrichment designs to optimize responders

• Evaluating the impact of expiry on drug requirements

• Determining the optimal timing of midstudy production runs and the quantity of medication required to commence the study

• Understanding the effect of enhanced flexibility in pack usage by letting the IVR system decide which combination of packs to allocate (e.g., 20 mg = 10 mg + 10 mg, or placebo + 20 mg).

Overall, however, we hope that this case study example has been able to demonstrate how simulation can be used by statisticians and supply chain experts to help ensure that adaptive designs are planned and implemented effectively. By doing this, study sponsors should have confidence in using these innovative designs and benefitting from their promise of more informative dose finding and accelerated drug development.

Graham Nicholls,* MSc, is product manager, randomization and supply chain management, and Bill Byrom, PhD, is product strategy director at Perceptive Informatics, Lady Bay House, Meadow Grove, Nottingham, NG2 3HF, UK, email: gnicholl@clinphone.com. Nitin Patel, PhD, is cofounder, chairman and chief technology officer at Cytel.

*To whom all correspondence should be addressed.

References

1. S. Frantz, "One in Five Drug Doses Set at Inappropriate Levels," Drug Discovery@Nature.com, September 2002, http://www.nature.com/drugdisc/news/articles/d060902-1.html.

2. G. Nicholls, D. McEntegart, B. Byrom, "The Adaptive Advantage," Applied Clinical Trials, June 2008, 102-108.

3. CytelSim Sytem Manual, Cytel Inc., Cambridge, MA, 2008.

4. M. Peterson et al., "Optimizing Clinical Trial Supply Requirements: Simulation of Computer-Controlled Supply Chain Management, Journal of the Society for Clinical Trials, 1, 399-412 (2004).

5. P. Bauer and M. Kieser, "Combining Different Phases in the Development of Medical Treatments Within a Single Trial, Statistics in Medicine, 18 (14) 1833-1848 (1999).

6. M. Posch, F. Koenig, M. Branson et al., "Testing and Estimation in Flexible Group Sequential Designs with Adaptive Treatment Selection," Statistics in Medicine, 24 (24) 3697-3714 (2005).

7. B. Byrom, "Managing the Medication Supply Chain Process Using Clinical Technology Solutions," Global Outsourcing Review, 7, 59-62 (February 2005), http://www.perceptive.com/news/articles/Managing_Med_Supply_Chain_GOR_feb_05.pdf.