Predicting Successful Site Performance

Choosing between placing a sizable bet on metrics to predict successful investigative site performance versus on implementing practices to support site performance success, most sponsors and CROs would choose the former. Ideally, both approaches play an important role. But most sponsors and CROs are investing in the former approach without much to show for it. Perhaps it's time that more sponsors and CROs focused attention on the latter?

Under tighter budgets and timelines, a number of pharmaceutical and biotechnology companies and CROs have been analyzing reams of descriptive and historical performance data supplied by investigative sites and maintained in house to identify factors that better predict successful patient enrollment. Four factors have emerged as promising predictors: past performance, experience, investigative site focus, and historic speed to randomize the first study volunteer.

Predictive factors

Retrospective analyses conducted by Pfizer and Lilly, for example, suggest that investigative sites that have performed well on one study are more likely to perform well on subsequent studies. Lilly's analysis in particular has shown that a comparatively well performing site on one study has a 70% higher chance of performing better than typical sites on future studies. Lilly also found that poor performing sites on one study have a high likelihood of under-performing in subsequent studies.

Investigator experience also appears to explain historical site success with some consistency. CROs and sponsors that have spent time analyzing site performance datasets have found that once a particular site has conducted six to 10 clinical trials, that site has a higher likelihood of meeting enrollment targets within the requisite time frame. Sponsors and CROs have concluded that investigative site operations move up a learning curve to achieve speed and effectiveness.

Pennsylvania-based RapidTrials, in collaboration with a researcher at the Waltham, MA-based Heller School at Brandeis University, have found that investigative site focus and commitment to conducting clinical trials is associated with higher performance. Based on data provided by 50 pharmaceutical and biotechnology companies on 72 Phase II-IV multi-center protocols across 10 therapeutic areas, RapidTrials found that investigative sites with sufficient infrastructure and focus on clinical trials were 41% better enrollers than sites devoting their time to clinical practice with limited attention to clinical research.

TTC, Inc. (also based in Pennsylvania), Merck, and Quintiles have each found that investigative sites quick to randomize their first patient tend to perform better overall. They argue that early enrolling sites know what they're doing. With a finite amount of time until data lock, early enrolling sites move through the contract and budget negotiations process faster, start enrolling patients sooner, and they leave more time to enroll patients throughout the study.

Taken at face value, these predictive factors appear promising in their ability to assist sponsors and CROs in identifying and selecting top enrolling investigative sites. Past performance, site experience, focus, and early enrollment speed each make good common sense and, arguably, have been intuitive and anecdotal considerations utilized by study monitors and site selection managers for some time. Sadly however, operating conditions and real world sponsor and CRO site management practices are rendering these predictive factors obsolete before they've had a chance to be applied.

Practice undermining prediction

The investigative site landscape, for example, is more fragmented today than ever. Of the approximately 23,000 FDA-regulated investigators conducting at least one clinical trial each year, only 35% are large, more experienced community-based sites and those affiliated with an academic health system; the majority are stand-alone sites conducting trials within a clinical practice. The investigative site landscape remains a cottage industry.

A growing proportion of investigative sites are novices—new entrants in the clinical research enterprise. According to CenterWatch, in 2000 two-thirds of all active investigative sites reported that they had been participating in clinical research for more than three years. In 2010, only 56% of active investigative sites reported doing so.

Turnover among this fragmented collection of investigative sites remains very high. CenterWatch has reported that four out of 10 US investigators on average, between 2000 and 2009, quit participating in clinical research each year up from 26% on average between 1990-1999. Investigative sites note that an onerous compliance burden, difficulties recruiting study volunteers, slow payment cycles, and the diminishing attractiveness of study grants have all contributed to the high drop out rate.

Sponsors and CROs are increasing the number of countries where they conduct clinical trials each year. The globalization of clinical trials is in large part motivated by sponsor and CRO efforts to lower clinical trial costs by engaging sites that will accept lower study budgets while offering more rapid patient recruitment rates. The Tufts Center for the Study of Drug Development (Tufts CSDD) has shown that between 2002 and 2006, sponsors conducted their Phase II and III trials in 11 and 19 countries on average respectively. Between 2007 and 2010, sponsors conducted their Phase II and III trials in 18 and 34 countries on average. Many of the sites engaged in these emerging markets are newer and less experienced entrants in the clinical research enterprise.

In an effort to diversify patient recruitment risk, sponsors and CROs have also been engaging larger numbers of investigative sites to enroll a smaller average number of study volunteers. This practice has forced companies to identify and select larger numbers of novice sites as many of the more experienced research centers cannot justify the high cost of committing resources and time to a study requiring only a few enrolled patients.

Proactive performance

The investigative site landscape is in a downward cycle, becoming ever more nascent and fragmented. Until it achieves scale and maturity, the site landscape will remain a moving target where metrics-predicted success will have limited value. Top performing sites will remain a scarce partner, trying to succeed in an operating environment that favors the lower cost alternative: larger numbers of less experienced investigative sites.

Will this cycle be broken? Can we ever expect to see more predictable site performance? Several trends portend a period when the investigative site landscape will consolidate and mature. I've written about the growing adoption of integrated outsourcing relationships which may help drive the establishment of long-term preferred CRO-site partnerships to better assure higher levels of study conduct performance and efficiency. In addition, several relatively well capitalized international site management organizations have signaled their interest in acquiring top performing investigative sites.

But predictable site performance need not only focus on the analysis of retrospective operating metrics. A growing body of research suggests that sponsors and CROs can positively affect site performance through improvements in protocol design; increased attention and support of site recruitment initiatives and operations management; and rising support of education and outreach initiatives.

Eight years of research that my colleagues and I have conducted at Tufts CSDD on protocol design complexity suggests a major opportunity for sponsors to ensure study conduct speed and efficiency. Our research has shown that more complex protocols are associated with poor recruitment and retention rates, longer cycle times, and larger numbers of protocol amendments. In turn, each protocol amendment requires on average two months of time and a conservatively estimated $500,000 to implement. Streamlined and simplified protocols, designed with feasibility input from study coordinators and investigators, would dramatically and positively impact the performance of investigative sites.

Quintiles recently reported success with the relationship managers it places in its select investigative sites. As part of its Prime Sites initiative, Quintiles found that a dedicated onsite manager keeps study staff focused and engaged in ensuring that select clinical trials receive the attention and resources required to meet timelines efficiently. A number of CROs have pointed to the importance of proactive investigative site support to ensure higher levels of successful performance.

CISCRP and other organizations have routinely shown the positive impact of outreach and education initiatives on patient recruitment and retention rates. In one study conducted in 2003 for example, systematic outreach and education—particularly to minority communities—increased the number of patients inquiring about participating in clinical trials by 63%. In 2007, Lilly ran a public service announcement concurrently in select markets where it was also recruiting patients for pain studies. The public education campaign never ran immediately before or after a specific recruitment ad. Rather, it was presented independently, airing at different times of the day, but targeting the same demographic populations as that of the patient recruitment advertisements.

Initially, Lilly found that investigative sites enrolled 38% more patients in markets where the public education message was airing concurrently. Over time, patients receiving the public education message showed a marked propensity to volunteer for Lilly's clinical trials. Eventually, enrollment rates per month doubled in those markets where the public education message aired concurrently.

Higher levels of successful site performance are within reach for all sponsors and CROs. Eventually, metrics-based approaches to routinely and consistently identify top performing investigative sites may become more effective. Until then, the best predictors of site performance success require hands-on proactive support including—but not limited to—the design of executable and feasible protocols; attention and support of site operations and recruitment; and ongoing public education and outreach.

Kenneth A. Getz MBA, is a Senior Research Fellow at the Tufts CSDD and Chairman of CISCRP, both in Boston, MA, e-mail: [email protected]