Protocol Appraisal: A Study Site's Viewpoint

There are many publications concerning critical appraisal of medical literature in the current push for the practice of evidence-based medicine.¹ These publications also address the issues of improving the reporting of clinical trial results² and of improving statistical aspects of trial design.³ There is, however, little published on the importance of a thorough review of clinical trial protocols to identify potential problems prior to their initiation, or on the issues important for a clinical trial site and its investigators when assessing whether or not to participate in an industry-sponsored study. In this review, we discuss issues that we have identified when appraising a protocol and propose a template for appraisal of trial protocols from the perspective of a clinical study site.

What is a protocol?

The design and performance of each experimental procedure in humans should be clearly formulated in an experimental protocol.⁴

A trial sets out to answer one or more questions or objectives that are presented as hypotheses amenable to statistical evaluation. The protocol states the objectives of the trial, provides a rationale for answering the proposed question, and describes the procedures that will be used to answer the study question. It should be detailed enough to allow the trial to be conducted by any appropriately qualified investigators. The protocol will be subject to regulatory and ethical review and should conform to relevant local regulatory standards and other guidelines. The typical protocol contents are listed in Table 1.

General considerations

When reviewing a protocol, it is worth considering the questions in Table 2. The scientific questions in this list deal with the merit of the trial hypothesis. The fundamental question of randomized controlled trials is, "Does the new treatment differ from the old treatment and, if so, how and by how much?" While the relative importance of the trial question is generally obvious, the real potential of a new treatment is difficult to assess objectively. Promising preclinical and early clinical data can lead to unrealistic expectations for a new drug, particularly when preclinical models are not a true reflection of

in vivo

biology. With accelerated drug development timelines, there is increased potential for indicators of drug failure being overlooked. Failure of the pivotal Phase III trials of gefitinib (AstraZeneca's Iressa) to demonstrate a benefit in the first-line treatment of nonsmall cell lung cancer is a classic example of a potential blockbuster failing to meet high expectations.

⁵

If trials aim to prove that a new treatment can improve on the existing standard of care, then a threshold for acceptance of the new treatment needs to be considered. This holds true for the increasingly common trials examining the equivalence of two treatments. Technically, statistical significance is important but for clinicians and patients, the magnitude of clinical benefit and the risk/benefit profile of the intervention are more important. A new cancer treatment that produces a statistically significant survival benefit of four weeks at the expense of excessive toxicity and impaired quality of life is not a major breakthrough. Investigators need to be clear in their own mind what constitutes a clinically important benefit and whether the proposed trial might be able to detect such a benefit. The difference between expected relative improvements and expected absolute improvements must be recalled. In a recent study of gabapentin for chronic headache, a 9.1% reduction of days with headache was reported.⁶ In practice this could translate to 1.5 days less each month; whether this is a clinically important and a cost-effective treatment is clearly debatable.

Table 1. Clinical Trial Protocol Contents, adapted from ICH Guideline for Good Clinical Practice7

Trial rationale

Medical research involving humans must not only conform to generally accepted scientific principles but also be based on a thorough knowledge of the scientific literature or other relevant sources of information, and on adequate laboratory or animal experimentation.

⁴

The rationale for undertaking a study must be clearly described. A good background section should discuss the reasons for undertaking the trial, and the potential adverse consequences of participation based on the currently available data. If this is not included in the protocol, the Investigators Brochure usually provides a more comprehensive safety review. The investigator must remember that for new drugs the overall amount of information on efficacy and safety is small and potentially subject to the biases of small sample size.

The control arm of a study should represent the medical standard of care and, therefore, the most ethically acceptable treatment. However, in some circumstances there may be no accepted standard of care due to a lack of prior evidence; variable patterns of care or differing opinions between institutions, countries, or individual practitioners; or the existence of apparently equivalent treatment options. While one could argue that the trial may establish the optimal treatment, if the standard comparator arm is unacceptable to the investigator it would be inappropriate for the investigator to undertake the trial. In the case of multidisciplinary interventions, all parties should agree that the trial treatments are acceptable, otherwise recruitment could be significantly compromised. Local practices may make adopting the protocol-defined standard of care impractical, particularly when the re-imbursement for the standard of care may vary substantially from country to country. This may result in study budgets being acceptable in one country but not another.

Trial design

Each stage of drug development raises different questions addressed by specific trial designs. Phase I trials assess safety and help determine treatment dose and schedule for subsequent studies, Phase II trials aim to assess efficacy in terms of response or a surrogate marker of response, and Phase III trials are usually comparative efficacy studies. Randomized Phase II trials are the most confusing. These studies usually serve one of two purposes; either determining likely efficacy compared to a comparator, or establishing the optimal dose or schedule for a new therapy. Information derived from Phase II studies is used to design subsequent Phase III trials. Phase IV trials also raise confusion: These trials may be observational studies or randomized trials and serve the purpose of expanding the information base related to a drug, whether from the perspective of efficacy, including new indications, or safety. Phase IV trials are often used as expanded access schemes for new medicines, and in our experience can cause problems related to ongoing drug supply at study termination and sometimes inadequate remuneration relative to workload.

Table 2. Template for critical appraisal of clinical trial protocols from the perspective of a clinical study site

The protocol should describe the randomization process and any processes to minimize systematic bias, as this can influence practical aspects of the study. The physical method of randomization needs to be clearly explained and reliable, especially when the randomization is centralized and accessed by computer or phone across multiple time zones. Blinding to the intervention received is an important method for minimizing bias. However, this may not always be appropriate or achievable, especially in the case of some intravenous therapies, where for safety and logistical reasons absolute blinding is sometimes impossible. Recently, our site declined participation in a trial as the investigators felt that the safety concerns related to blinding of a potentially toxic intravenous agent outweighed the importance of the study question.

The method of administration of treatment is generally well described, but often the guidelines for treatment modification are insufficiently detailed and can lead to unexpected protocol violations. The best way to determine the usability of the treatment instructions, and the protocol in general, is to consult the users, such as trial coordinators, nursing staff, and pharmacists. Clarifying these concerns with the sponsor prior to study initiation will often resolve the issue, sometimes through protocol amendment.

Study population and eligibility criteria

Eligibility criteria serve to identify the target population and exclude patients that potentially could be harmed by the treatments or confound the results of the study. Investigators should carefully consider the target patient population and the expected accrual rate at their own site. Logistically, it may not be sensible to undertake a trial if the site will not be able to accrue a reasonable number of patients. Clinical trial subjects are not reflective of "reality"

^8,9

and restrictive eligibility criteria are probably the greatest contributor to poor trial recruitment. Recent surveys have suggested that up to 95% of an otherwise eligible target population is not eligible to participate in industry-sponsored trials.

⁸

A common mistake is to take into consideration the therapeutic indication while failing to adequately consider the exclusion criteria. A patient with angina is not the same as a patient with angina who must be a male age 45-70 with type II diabetes and hypertension.

Assessment of efficacy and toxicity

The protocol must clearly state the primary and secondary outcome measures of efficacy, and how these will be evaluated. The primary outcome measure forms the basis for the statistical analysis and sample size calculations. Sample size calculations are based on the variance in an outcome measure and the expected magnitude in differences between treatments. The sample size calculations are often only valid for the primary outcome measure. This in itself is not a major flaw in study design, but from a practical perspective investigators should be aware that more secondary endpoints translates into increased work for the study team.

Familiarity with study endpoints and measurements is important. Some investigations may have limited availability or high cost that may limit feasibility of the trial at a site. In some situations, specific training or accreditation needs to be obtained in order to perform the study measurements. The invasiveness of the procedures and convenience for the study participants must also be considered, as this will have great impact on study accrual and compliance.

Ethical considerations

The research protocol should always contain a statement of the ethical considerations involved.

⁴

Many of the ethical considerations in clinical trials can be resolved if the scientific and medical rationale for the study and the scientific method used to implement the study is sound. An ethical issue previously mentioned is ensuring that an appropriate standard of care is used. Unfortunately, there is an increasing dilemma for trialists as effective treatments exist for an increasing number of medical conditions. For example, rheumatoid arthritis trials frequently require subjects to have failed several disease-modifying therapies, have active disease, but still be on maintenance methotrexate. Despite the presence of active disease, the eligibility criteria may restrict recruitment by imposing additional criteria, e.g., elevated ESR or C-reactive protein, that may not be met due to the methorexate therapy.⁹

With respect to Phase I studies, there is a unique ethical problem. The Declaration of Helsinki states that "medical research is only justified if there is a reasonable likelihood that the populations in which the research is carried out stand to benefit from the results of the research."⁴ In the context of Phase I studies, there is often a very low or even negligible chance of benefit, and patients should be informed of this in the consent document and during the accompanying investigator-patient discussion. This issue is further complicated in the setting of healthy volunteers, where there is no medical benefit and the only incentive might be financial.

The Declaration of Helsinki states "at the conclusion of the study, every patient entered into the study should be assured of access to the best proven prophylactic, diagnostic, and therapeutic methods identified by the study." Ethically, this is fair, but pragmatically it is often not achievable, and has been an increasingly common problem at our site. For early phase studies there simply may not be enough study drug available to provide ongoing supply, especially in the case of biopharmaceuticals. Furthermore, given that a Phase I study does not establish proof that a treatment is effective, continuing supply of drug should not necessarily be considered in the decision to participate in a study.

In later stages of development, local considerations often determine the availability of study drug. For example, in Australia there is a variable period between registration and the governmental decision to subsidize the cost of the drug (if at all). During this time, companies may no longer provide study medication free of charge to the study participant. Consequently, study patients may need to stop the drug regardless of its effectiveness, and make alternate arrangements or, in some cases, the hospital drug committee may be morally obliged to cover the drug costs. However, hospitals are increasingly unable to provide such assistance. Investigators should clarify with sponsors the arrangements for ongoing supply of drug prior to participating in a study.

Logistical considerations

Trial management consumes vast resources in terms of time, money, and manpower. If a site accrues too many patients or has too many trials, site resources may be stretched too far, compromising both trial quality and patient safety. If only one or two patients are accrued to a trial, it may simply be a waste of resources that could be diverted elsewhere. These difficulties for sites are compounded by irregular volumes of activity and payment schedules geared toward payment at the completion of subject participation. Furthermore, "completion" is defined as that time at which the monitor has visited the site to ensure that all documentation is complete—a somewhat tenuous timeframe that is outside the control of the study site. From our perspective, it is probably safer to aim to have enough studies so that activity is likely to be slightly higher than capacity, rather than to aim to meet capacity and fall short due to underrecruited studies.

Factors influencing recruitment include the eligibility criteria of the trial, the availability of the patient population, the expected overall accrual timeline of the trial, and acceptability of the trial to patients. Databases may facilitate assessment of anticipated numbers accruable but these still tend to over-estimate the eligible population. When sites are approached to join a study well after other sites have started accruing, it is a sure sign that it will be difficult to recruit to the study. The referral base is another important factor: surgeons or physicians who will be relied on to refer patients must be aware of the trial and preferably have a track record for referrals rather than just well-meaning enthusiasm. In the situation where there are multiple or external investigators, at least one investigator should take a genuine leadership role to promote recruitment. Trial complexity is a major consideration. Frequent or long visits, or multiple procedures, can be a major disincentive to participation, especially in the case of healthy volunteer studies or for very unwell patients.

The practical aspects of coordinating the increased demands of clinical trials for a sick patient population also require careful consideration. A recent study of a treatment for motor neuron disease at our site required early morning visits in order to complete the study assessments that were conducted at multiple clinics, and to accommodate the schedule of the busy and limited clinic times. Given the travel difficulties, preparation times, and reliance on carers that is characteristic of this patient population, many of the participants found the visits too onerous and opted to withdraw prematurely.

Assessing the study coordinator workload associated with a trial is difficult. If possible, it is worthwhile to obtain a copy of the case report form (CRF), as this will indicate the amount of data to be collected and collated. Some trials take a minimalist approach to data collection and are "quick and simple and reliable."¹⁰ Most trials, however, examine multiple endpoints, and the data collection requirements are substantial. This is particularly the case for industry-sponsored Phase III studies aimed at drug registration. These trials will increasingly incorporate additional time-consuming data collection including patient diaries, quality of life, economic health assessments, and additional samples for population pharmacokinetics and pharmacogenetics. As trials become global, further logistic considerations include remote randomization and shipping tissue, blood, or imaging to other centers for central processing or independent assessment. As investigators, we tend to forget that each hour of patient contact may generate several more hours of documentation activities for our study coordinators. Finally, and importantly, not to be underestimated is the time spent by study staff with monitors, auditors, and at investigator and initiation meetings. A recent survey reported that a standard oncology trial required approximately 200 hours per subject and that 32% of this time was devoted to nonclinical time.¹¹

Funding for a trial should be at least adequate to cover the costs of trial administration, otherwise the center will operate at a loss. Many believe that all of the costs of clinical trials should be remunerated, especially in the case of industry-sponsored trials. A common practice, however, is to reimburse only the trial costs associated with per-protocol procedures that are not part of standard care. This might be acceptable when the site is financially supported by a parent institution, e.g., free use of space in a hospital, but for a fully independent site such trials are financially unsatisfactory. Staff commitments to the project management and trial documentation are a major source of study cost, and are often underrepresented in the budget. Other items that are often neglected in study budgets include the cost of preparing ethics applications and the archiving of data once the study is closed. Itemized budgets should be requested from the sponsor where possible. If the proposed study payment is too low, then submit an itemized budget to the sponsor to clarify and justify the need for an increased budget.

A major burden in our unit is expanded access schemes or extension studies from previous trials. As indicated previously, these schemes provide the opportunity for ongoing drug availability at the end of a trial. Unfortunately, in our experience, these schemes are remunerated less than the original studies and require as many or sometimes more resources.

Sponsors often request feasibility assessments well prior to a trial commencing. In our experience, the site is often provided with a very limited description of the protocol, and for confidentiality reasons the name of the study drug is usually withheld. This decreases the ability of the site to provide adequate feedback regarding the protocol, both from the perspective of feasibility (including budgetary estimates), and level of interest. A confidentiality agreement may overcome these problems. If the feasibility review is unsuccessful, feedback from the sponsor should be requested so the relevant issues can be addressed for interactions with future sponsors.

Protocols are commonly supplied to investigators relatively late in the start-up period of a trial, and the majority of sites will have little input into the protocol. Input from a site is more likely if the trial is early phase. Regardless of this, we believe it is beneficial to provide to the sponsor with some feedback regarding any problems related to a protocol. The benefit of this is that sometimes appropriate amendments can be instituted; in addition, the design of future studies may be improved, as will the timelines for getting important new treatments to patients who need them.

Strategic considerations

Given the resources required to establish and maintain a clinical trials unit, the motive for establishing a site should be defined. Reasons may range from enthusiasm for clinical research to purely financial incentives. Our main priorities are serving our academic interests, contributing to improvements in the standard of care of patients, and providing a service to our affiliate institutions. The motivation for establishing a site should be considered when selecting a trial. Otherwise, participation will be a futile and unrewarding experience.

The expertise and academic interests of our staff may influence direction of the unit. The identification of areas of therapeutic expertise and establishment of niche specialities, e.g., human experimental models, is an important aspect of developing a research strategy and business plan. Strategic alliances and professional networks help define the nature of the trials the unit adopts. Specific academic partnerships and partnerships with industry can lead to a "pipeline" of trials. While not always relevant to small sites contributing to large studies, it is nonetheless important to ensure that appropriate publication rights are in place, as publications contribute to a track record for the site.

Conclusions

In this article we have outlined some of the issues that we believe a clinical study site should face when considering whether to undertake a study. Systematic appraisal of the protocol prior to agreeing to participate is an important process to allow problems to be identified, and to ensure that the trial site has made a considered and informed decision to participate. This process is important because overall the clinical research process can be improved by sites taking a proactive role in the design and conduct of clinical trials.

References

1. G.H. Guyatt, "Users' Guides to the Medical Literature [Editorial],"

JAMA,

270 (17) 2096-2097 (1993).

2. D. Moher, K.F. Schulz, D.G. Altman, "The CONSORT Statement: Revised Recommendations for Improving the Quality of Reports of Parallel-Group Randomised Trials," Lancet, 357 (9263) 1191-1194 (April 14, 2001).

3. D. Moher, C.S. Dulberg, G.A. Wells, "Statistical Power, Sample Size, and Their Reporting in Randomized Controlled Trials," JAMA, 272 (2) 122-124 (July 13, 1994).

4. World Medical Association Declaration of Helsinki Ethical Principles for Medical Research Involving Human Subjects, sourced online at http://www.wma.net/e/policy/b3.htm on 2/December/2003.

5. J. Dancey and B. Freidlin, "Targeting Epidermal Growth Factor Receptor—Are We Missing the Mark?," Lancet, 362 62-64 (2003).

6. P. Spira and R. Beran, Australian Gabapentin Chronic Daily Headache Group, "Gabapentin in the Prophylaxis of Chronic Daily Headache. A Randomized, Placebo-Controlled Study," Neurology, 61, 1752-1759 (2003).

7. ICH Topic E6 Guideline for Good Clinical Practice, sourced online at http://www.emea.eu.int/pdfs/human/ich/013595en.pdf on 2/December/2003.

8. S.R. Tunis, D.B. Stryer, C.M. Clancy, "Practical Clinical Trials: Increasing the Value of Clinical Research for Decision Making in Clinical and Health Policy," JAMA, 290 (12) 1624-1632 (September 24, 2003).

9. T. Sokka and T. Pincus, "Eligibility of Patients in Routine Care for Major Clinical Trials of Anti-tumor Necrosis Factor Alpha Agents in Rheumatoid Arthritis," Arthritis Rheum., 48 (2) 313-318 (February 2003).

10. QUASAR Collaborative Group, Comparison of Flurouracil with Additional Levamisole, Higher-dose Folinic Acid, or Both, as Adjuvant Chemotherapy for Colorectal Cancer: A Randomised Trial, Lancet, 355 (9215) 1588-1596 (May 6, 2000).

11. E. Emmanual, L. Schnipper, D. Kamin, J. Levinson, A. Lichter, "The Costs of Conducting Clinical Research," J Clin Oncol, 21, 4145-4150 (2003).

*To whom correspondence should be addressed.

Acknowledgements

The authors would like to thank Rodney Ecclestone, Executive Officer, St Vincent's Hospital, for frequent candid discussions related to many of the problems highlighted in this article.