Increasing Study Sensitivity in Early Trials

Patient Reported Outcomes (PRO) are becoming ever more important in drug development and marketing, driven in part by the need to demonstrate value to the patient, who is the ultimate "customer" for drug development. The technology for collecting PRO data has lagged behind the electronic age, with many investigators still using paper diaries. However, concerns about recall bias¹ and the validity of paper diaries are turning investigators toward electronic diaries. The primary benefit of eDiaries lies in its ability to capture PRO data in real time, which increases study sensitivity, i.e., the ability to detect real drug effects.

However, many researchers wait until the late stages of clinical development to introduce eDiaries. In this article, we review why eDiaries are important in Phase II trials, delivering a greater ability to detect treatment effects and reducing the size and costs of drug development programs.

Learn vs. confirm

The goal in a Phase II trial is to assess the viability of a candidate for progression to late stage development. This includes identifying the correct dose and perfecting methods for Phase III testing. Since development costs rapidly increase once a compound enters Phase III testing, the pressure on a Phase II trial is to determine if the drug works and at what optimal dose or dosages—to inform the decision whether to invest in a Phase III trial.

In a recent Harvard Business Review publication,² experimenters from Eli Lilly suggest that when development costs are high and failure is common—as they are in new pharmaceutical development—companies can reduce development risks and costs by taking fundamentally different approaches to how they structure their early and late stage research. They recommend a truth-seeking early stage, focused on evaluating novel products' prospects and eliminating bad bets, and a success-seeking late stage, focused on maximizing the value of products that have been cleared for development.

These concepts draw attention to the basic principles of early vs. late stage pharmaceutical research. Phase II is intended to be the time for clinical development teams to learn about the drug and explore its properties. The first imperative is to inform a go/no-go decision about the drug before further research dollars are spent. If a drug is to proceed, the second imperative is to better understand the drug, so that successful Phase III trials can be designed. It is only during Phase III testing—the confirmation stage—that researchers should focus their efforts on demonstrating the drug's effectiveness to others (regulators and payers) and pursuing marketing success.

One could argue that the demands of learning about a drug's effectiveness are actually higher than the demands of demonstrating them to an external audience. This is why sponsors should focus on obtaining as much data as possible and ensuring the highest data precision throughout all aspects of Phase II clinical testing.

PRO strategies in Phase II

Although eDiaries have been widely adopted for capturing PRO data during Phase III testing, most researchers have not fully recognized their benefits during early development. This may be because researchers tend to look at Phase II development as "exploratory" and because they are not subject to as much regulatory scrutiny as a Phase III trial, perhaps not requiring the best methods. There is also the bottom line rationale: Smaller Phase II trials don't offer the same economies of scale that Phase III does. That is, the per-subject costs of developing an eDiary in Phase II are often higher than in Phase III, merely because fixed costs can not be spread out over as many patients.

The case for eDiaries

The advantages of eDiaries for the collection of PRO data are well documented.³ In addition to gaining significant operational efficiencies, researchers benefit from valuable scientific improvements that eDiaries offers. By capturing PRO data in real time, eDiaries overcome the recall biases associated with back-filled paper diaries. By also gathering multiple data-points, eDiary data can provide a more detailed and reliable understanding of the drug's effects.

The greatest benefit of real time data comes from improved study sensitivity, which translates into more informative studies. The methodological improvements provided by an eDiary solution can help squeeze out "noise" that pervades paper diary data (and nondiary PRO data). Reduced noise translates into increased statistical power—the ability to detect real drug effects. So, to put it simply, the more you can squeeze noise out of your measurement, the more powerful your test is going to be.

There are a number of examples of pharmaceutical researchers who have improved study sensitivity by collecting real time PRO data with eDiaries. The first example is a Phase III trial of a selective serotonin and norepinephrine reuptake inhibitor (SNRI) medication for fibromyalgia, in which the sponsor, Cypress Bioscience, collected pain data, which was one component of their composite endpoint, using eDiaries. Paper reports of pain intensity were also captured at site visits. Therefore, two different vantage points on patients' pain perceptions could be compared—daily pain assessments captured on the eDiary, and in-clinic self-reports of pain over the past 24 hours and seven days prior to site visits captured using traditional paper measures.

The reported effect sizes (a standardized quantification of the effect of the drug, accounting for variability) for the approved 200 mg/day dose are striking. In one clinical study, the drug effect relative to placebo, as assessed by pain reports on the eDiary, was 2.3 times greater than as assessed by paper-based pain assessments at office visits. Indeed, while the eDiary measure showed a statistically significant drug effect in this study, the parallel paper-and-pencil reports did not.4 This illustrates the implications for studies' sensitivity to drug effects.

Besides being considered and computed retrospectively, as above, measurement sensitivity and its effect on the expected effect size also have prospective implications for the sizing of subsequent studies in a drug development program. Using more sensitive methods and measures may allow for smaller studies to detect drug efficacy. Our calculations show that increasing the effect size by a factor of 2.3 can reduce the sample size needed for detecting the effect by at least 80%. For example, as illustrated in Figure 1, in a hypothetical study where the standardized drug treatment, as assessed by eDiary, was 0.125 (that is, an eighth of a standard deviation difference between active and placebo), an eDiary study would require 1006 participants to reliably (80% power) detect a treatment effect, whereas a paper-and-pencil-based study would require 5435 participants to reach the same power.

Of course, sample size may be driven by other considerations, such as meeting ICH guidelines or the evaluation of safety-related endpoints, but the argument remains: a more sensitive measure is more likely to detect drug effects, and to do so with smaller samples.

Another dramatic example involved a Phase III study of a treatment for overactive bladder in which the primary efficacy endpoint included a count of the patients' daily number of micturitions. Compared to an earlier, similar study conducted with paper diaries, an eDiary solution reduced data noise (error variance) by 33%.⁵ The sponsor, Sepracor, translated the 33% drop in error variance into a 50% decrease in the number of patients that would have been needed to detect drug effects.

An interesting component of the Sepracor example is that instead of measuring symptoms, the PRO data represented a count of events. Research has shown that when you ask someone to count anything, they generally tend to round up or down. This is common when you're asking for a count of events, which people tend to round to increments of five or 10 or when you ask for a time, in which responses are typically rounded to quarter, half or one hours. While this might not matter in everyday conversation, it can significantly degrade data precision and seriously mask treatment effects in a clinical trial.

Take, for example, a smoking cessation study which assessed how many cigarettes subjects smoked per day during treatment. Subjects engaged in real-time tracking on an eDiary, and also reported their cigarette consumption retrospectively, using timeline follow-back methods. Recall data were heavily subject to rounding, with most numbers listed as 20, 30, 40, and 15 respectively, which introduced substantial error.⁶ The eDiary data showed no "digit bias" at all, capturing the end-point precisely.

These tangible examples demonstrate how collecting PRO data in real time using an eDiary enables sponsors to capture more precise data, which means less noise, greater power, smaller studies (i.e., faster and cheaper) and a better ability to detect clinical effects.

Study sensitivity

What is the benefit of eDiary's enhanced study sensitivity for Phase II studies? First, because Phase II studies are often smaller than Phase III studies, the ability to detect drug effects with smaller samples becomes crucial. Earlier, we posited that the strategic pay-off of Phase II lay in dropping drugs that are not promising. However, if such no-go decisions are to be made, they need to be made with confidence. An under-powered study can lead to false-negative results that might result in a promising drug being dropped. Conversely, by providing more definitive information, more sensitive studies can provide the information a sponsor needs to "kill" an unpromising drug with confidence that it is making the right decision.

The sensitivity of eDiary data is not just useful for go/no-go decisions. It is essential for dose-selection decisions for a drug that is to go forward. Demonstrating that a drug is superior to a placebo can be difficult, and eDiaries can help. In Phase II, dose-ranging studies present an even steeper challenge: distinguishing among doses of the active drug. Study sensitivity becomes even more critical in these contexts, and can help sponsors select the right dose of the drug to bring forward into pivotal testing in Phase III.

Finally, if the candidate drug is entering a crowded category with established competitors, it is often important to demonstrate superiority, or at least parity, with the established product. This too requires maximum study sensitivity.

Continuing to Phase III

During the Phase II learning stage of research, sponsors have an opportunity to fully understand the details of the disease and drug, including the time course of symptoms and treatment effects, which could have significant impact on the optimal design of later-stage trials. If used appropriately, real time PRO data captured at the time of the experience can produce a much more precise profile of the symptoms and drug effects and help researchers to better focus their Phase III assessments.

For example, eDiary data collected throughout the day from patients with rheumatoid arthritis⁵ show that pain varies dramatically over the course of the day, peaking in the morning and dropping dramatically at night (see Figure 2). This knowledge informs the approach to assessing the impact of medication, suggesting that the assessment in a Phase III trial must take care to include morning pain, lest an assessment at other times or a less-specific all-day pain summary (such as the commonly used end-of-day assessment) mask the patient's pain experience, and thereby also hide the drug's effects on pain.

Phase II should be the time for researchers to prepare for Phase III in other ways. Phase II is a key opportunity to develop and validate the PRO measures that will be used in Phase III trials. Especially when the end-points being assessed vary with time, as many do, this may require a retooling of the measurement approach, for example, shifting focus from big evaluations of broad quality of life metrics to really getting detailed measures of symptoms, and to focusing the timing of measurements on current state vs. global recall, all of which could have significant impact on the trial's endpoints and product claims. Phase II provides an opportunity to validate novel eDiary approaches in preparation for confirmatory Phase III studies.

Besides the science, the team and the processes also need to be developed in Phase II.

Introducing ePRO in Phase II builds the experience and capabilities of the team and allows the team to develop processes that will facilitate the Phase III work. Those who question the economies of scale in Phase II work with eDiaries may fail to consider the economies of preparation—the ability not only to reuse eDiary programming and equipment, but also to embark on Phase III with well-honed processes and an experienced team.

Conclusion

eDiaries can have significant impact on data quality and integrity, increasing study sensitivity in ways that favorably impact drug development economics and timelines. Developing a comprehensive PRO strategy that guides a program from early through late stage development, clinical teams can undertake early adoption of methods that can help them gain greater insight into the full therapeutic potential of their candidate drugs. Sponsors who take this approach will recognize that the data demands of learning about a drug in Phase II are greater than the data demands of confirming those learnings in later stage testing and, as such, will benefit from the most reliable forms of data capture, such as eDiary.

Saul Shiffman, PhD, is the Co-Founder and Chief Science Officer of invivodata, Inc., 2100 Wharton Street #505, Pittsburgh, PA 15203, email: [email protected].

References

A.A. Stone, S. Shiffman, J.E. Schwartz, J.E. Boderick, M.R. Hufford, "Subject Non-Compliance with Paper Diaries," British Medical Journal, 324, 1193–1194 (2002), and S. Shiffman, A.A. Stone, M. Hufford, "Ecological Momentary Assessment," Annual Review of Clinical Psychology, 4, 1-32 (2008).

E. Bonabeau, N. Bodick, R. Armstrong, "A More Rational Approach to Product Development," Harvard Business Review, March, 2008, 96-102.

M.R. Hufford and S. Shiffman, "Methodological Issues Affecting the Value of Patient-Reported Outcomes Data," Expert Review of Pharmacoeconomics and Health Outcomes, 2 (2) 119-128 (2002).

D.A. Williams, R.M. Gendreau, D.J. Clauw, Electronic Diaries have Superior Discrimination Compared to Paper-based Pain Assessment in Individuals with Fibromyalgia. Presented at the 71st Annual American College of Rheumatology and 42nd Annual Association of Rheumatology Health Professionals Meeting, November 6-11, 2007.

S. McKenzie, J. Paty, D. Grogan, M. Rosano, L. Curry, H.M. Sciarappa, "Proving the eDiary Dividend," Applied Clinical Trials, June 2004.

S. Shiffman, "How Many Cigarettes did You Smoke? Assessing Cigarette Consumption by Global Report, Time-Line Follow-Back, and Ecological Momentary Assessment," Health Psychology, 28 (5), 519-526 (2009).

A.A. Stone, Unpublished data on 105 rheumatoid arthritis patients sampled from community medical practices. Patients completed 5 to 6 assessments per day for one week.