Enhancing Trial Enrollment Through 'Goal Programming'

Clinical trials often fail to reach desired goals due to poor recruitment outcomes, including low participant turnout, high recruitment cost, or poor representation of minorities. At present, there is limited literature available to guide recruitment methodology. This study, conducted by researchers at the University of Wisconsin Center for Tobacco Research and Intervention (UW-CTRI), provides an example of how iterative analysis of recruitment data may be used to optimize recruitment outcomes during ongoing recruitment.

Study methodology

UW-CTRI's research team provided a description of methods used to recruit smokers in two randomized smoking cessation trials (n = 196 and n = 175). The trials targeted low socioeconomic status (SES) smokers and involved time-intensive smoking cessation interventions. Primary recruitment goals were to meet required sample size and provide representative diversity while working with limited funds and limited time. Recruitment data was analyzed repeatedly throughout each study to optimize recruitment outcomes.

Results: Estimates of subject recruitment outcomes based on prior studies targeting smoking cessation suggested that researchers would be able to recruit 240 low SES smokers within 30 months at a cost of $72,000. With employment of methods described herein, researchers were able to recruit 374 low SES smokers over 30 months at a cost of $36,260.

Discussion: Each human subjects study presents unique recruitment challenges with time and cost of recruitment dependent on the sample population and study methodology. Nonetheless, researchers may be able to improve recruitment outcomes though iterative analysis of recruitment data and optimization of recruitment methods throughout the recruitment period.

Recruitment time, number, and cost

The success of any human subjects trial depends on the enrollment of a sufficient number of participants. Unfortunately, in the majority of human subjects studies, recruitment takes longer than expected and often fails to provide a representative sample of the target population. ^1,2,3,4,5 One-third of publicly funded trials continue to require a time extension due to failure to meet recruitment goals,¹ and investigators who have been unsuccessful recruiting the requisite number of participants to power their studies are often forced to apply for additional funds if they wish to obtain publishable results.^6,7,8

A review of extant literature shows that recruitment costs vary widely based on recruitment method. Kolawole S. Okuyemi conducted a study to determine the efficacy of nicotine gum and health counseling to help low-income African American smokers quit smoking, and spent $156 per enrollee via directed mailing, but $5,040 per enrollee via gas-pump advertisement.⁸ Other studies on smokers with recruitment data available showed cost per enrollee ranged from $18 to $193.^6,9 With a wide range of recruitment costs and many factors affecting these costs, it is important for researchers designing a study to choose their recruitment methods knowledgably.

Representative sampling

Representative minority sampling is a well-established goal in human subjects recruitment, and under-representation of minority subjects or low SES subjects is a problem in many trials.^{10,11,12,13,14,15} Recruitment of minorities compared to non-minorities has been shown to result in lower total enrollment rates and higher cost per enrollee.^{16,17,18,19,20} Individual methodologies for recruitment of minorities vary from study to study, but common strategies include advertisement in regions with high-minority representation, use of bilingual recruitment information, and community-based participatory research.^{8,9,21,22,23,24} While minority targeting is important in order to obtain a desired number of minority subjects, literature on strategies to optimize minority recruitment outcomes is scarce.

A problem with recruitment

To UW-CTRI's knowledge, there are presently no examples in the literature of how researchers might best conduct analyses of ongoing recruitment data to optimize recruitment outcomes.²⁵ Most researchers at large universities do not use a formal system to optimize recruitment outcomes. Instead, they typically create the design and budget for study recruitment based on a prior study with similar population and study design. Then once recruitment has begun, researchers leave these recruitment strategies in place without substantial evaluation. This means that poor outcomes often go unrecognized, or if they are recognized, there is no system in place to use this knowledge to implement meaningful modifications in recruitment strategy.

Goal programming

UW-CTRI's proposed method of iterative optimization is derived from a similar process used in business called "goal programming." Goal programming refers to a process whereby collected data is assessed via a mathematical algorithm that prioritizes various outcome goals.^26,27,28 By repeatedly analyzing and assessing emerging data against outcome goals, a business is able to make intelligent modifications of various business strategies such as advertisement, targeted population, or expense allocation.²⁶ An example of this comes from the corporation Google, which recently published results of an advertising campaign in which it used goal programming to substantially reduce expenses while maximizing returns.³⁰ In many research projects that are small and run on limited funds, it would not be cost- or time-effective to develop sophisticated computer programs to assess recruitment outcomes. The general model, however, might be applied to research and used widely if it remains low-tech and cost-effective. In this study, researchers followed the basic methodology of goal programming used in business, but applied simple, non-computerized analytical methods that might be replicated at low expense in most human subjects trials. The steps of this method include development of a starting strategy, goal assessment, data collection, iterative analysis, and strategy optimization.

Steps used to establish and modify recruitment strategy: This section will further describe the aforementioned methodology.

• Goal assessment: Develop a conceptual algorithm that prioritizes recruitment goals, identifies acceptable limits for each goal, and integrates conflicting goals.

• Data collection: Prior to starting recruitment, design a recruitment data collection system that includes information for each caller and the type and location of advertisement encountered.

• Iterative analysis: Repeatedly generate a recruitment report (for example, every four weeks) on the previous recruitment period, including information on time, cost per enrollee, and demographic distribution of enrollees. Conduct a simple analysis of each recruitment method (flyers, radio, television, etc.) and recruitment time period so that data from each recruitment method and successive iteration may be compared.

• Strategy optimization: Employ a goal-based algorithm to make a modification in recruitment strategy that best serves the recruitment goals of the trial.

Goal programming implementation

UW-CTRI used this method of iterative optimization to maximize recruitment outcomes in two smoking cessation studies, Mindfulness Training for Smokers 1 and 2 (MTS1) (MTS2). Both studies were NIH-funded, randomized controlled trials that evaluated the efficacy of a seven-week mindfulness training program for smoking cessation targeted to low SES smokers. The researchers' initial assessment study involved testing the method on the first study (MTS1). However, they were also able to include the MTS2 study due to favorable results.

Step 1) Starting strategy: Through determination of effect sizes and calculation of number-needed-to-treat, UW-CTRI determined that it would need to recruit 240 participants within 30 months. Researchers found that in previous similar smoking cessation studies from the lab, the cost per enrollee fell between $300 and $900. These studies relied primarily on television advertisements, which had been determined to be the most cost-effective means of recruitment. These studies did not target low SES participants or provide intensive interventions as UW-CTRI's study did, and, therefore, it was thought that recruitment in this study might be more costly. Nonetheless, researchers optimistically set the initial cost estimate at $300 per enrollee, with a total cost for recruitment calculated at $300 x 240 participants = $72,000. The recruitment drive was divided into twenty 22-day "waves" with a goal of receiving 48 callers per wave. The first wave of the trial used only TV, because this method had proven most effective in the past.

Step 2) Goal assessment: Recruitment goals were identified and prioritized.

• Supply the study with a sufficient number of participants to reach statistical significance between groups (240 subjects).

• Remain within recruitment budget of $72,000.

• Recruit a representative sample of smokers, including a larger portion of low SES smokers than is normally found in the area.

• Diversify recruitment methods to minimize the risk of experiencing market saturation and to maximize the possibility of discovering cost-effective methods.

Step 3) Data collection: The UW-CTRI team used phone screening questionnaires to collect information from callers on the type of ad, time and place of ad encountered, smoking history, gender, age, ethnicity, and education level. This information was organized in such a way that recruitment methods could be compared.

Step 4) Iterative analysis: At the completion of each recruitment wave, a report was generated with the number of callers, number of enrollees, demographic divisions, and cost per enrollee. Statistical analyses were all conducted via Microsoft Excel and primarily required construction of graphs and diagrams with comparison of means via t-tests. The report took an undergraduate research assistant approximately six hours to compile and was presented to staff at the end of each wave (every seven weeks) at a lab meeting.

Step 5) Strategy optimization: After each report, research staff discussed which research methods were most or least costly and most or least effective, and implemented a single modification that might improve outcomes the most. For example, after one wave, data might show TV advertisements to be significantly more expensive than other forms of advertisement, and this might lead to the decision to decrease spending on TV. Researchers only made one modification per wave so as to best isolate and understand the effect of the modification during future analyses.

Results

Researchers initially anticipated that MTS1 would require 240 participants for sufficient power, with a 30-month recruitment period and a $72,000 budget. After several waves, researchers observed a larger-than-expected effect size between groups in smoking cessation, the primary outcome. When researchers recalculated the number-needed-to-treat based on this larger effect size, they found that they would need only 180 participants in order to achieve statistical significance and complete the study. With iterative analysis of incoming data, and on-going modification of recruitment strategies, researchers were able to recruit the desired 180 participants within 11 months at a cost of $18,482. This savings in time and funds allowed for the addition of a second study. The MTS2 study reached a goal of 175 enrollees over 15 months at a cost of $17,778. Thus, from a budget originally intended to enroll 240 participants at a cost of $72,000, the research team was able to enroll 376 participants at a cost of $35,215. The average number of calls generated in each recruitment wave = 121.7 (SD = 26.9, target = 48); average enrollment per wave was 22.5 (SD = 7.6, target = 12); average cost per enrollee was $94.16 (SD = $33.37; target = $300/enrollee). Excluding "word of mouth," flyer placement—including the cost of materials, fuel, and research assistant time—provided the lowest recruitment cost per enrollee, followed by newspaper and TV advertising (See Table 1).

The following are descriptions of how iterative analyses were used within a goal-based algorithm to develop modifications to recruitment strategy. The modifications described are not intended to be exhaustive, but instead to provide an understanding of the type of decision-making employed.

Modification 1: TV was used exclusively during the first wave because of its historical success at UW-CTRI's center. Initial recruitment data showed that TV advertisement costs were $260 per participant (closer to the $300 found in prior studies). Because of this, researchers decreased funding on TV and allocated funding toward flyer postings and newspaper advertisements. The introduction of these two methods resulted in a substantial increase in enrollment for Wave 2 and a decrease in overall cost.

Modification 2: Recruitment in Wave 2 (MTS1) yielded groups that were actually larger than needed. Assessment of this data against the team's goals yielded the conclusion that flyers substantially reduced cost per enrollee, though TV and newspaper continued to have high cost and provide poor enrollment. Based on this information, UW-CTRI eliminated newspaper advertising in Wave 3. Researchers kept TV, because historically this method had been most cost effective for this population (See Figure 1).

Modification 3: By Wave 4 (MTS1), it became clear that TV advertising continued to be less cost-effective than other methods and that overall recruitment through TV was decreasing. With this, researchers felt that TV was proving to be consistently more expensive than other methods, so they reduced spending on TV and reintroduced newspaper advertisement. Newspaper was reintroduced in order to maintain diversification.

Modification 4: Newspaper advertisement was eventually discontinued after a sustained period of decreasing effectiveness. Researchers suspected that this might represent market saturation; that over time, smokers who read that newspaper had already responded to the advertisements. Because of this trend, advertisement in one periodical was replaced with that in a different publication.

Discussion

In the studies presented, modifications made based on iterative optimization appear to have improved recruitment outcomes. This method allowed researchers to identify the relative success or failure of each recruitment method and make informed modifications. Although in many cases, post-hoc analyses of recruitment response did not reach statistical significance (i.e., comparison of mean cost per enrollee for newspaper vs. flyer yielded p = .30), differences were large enough to warrant modifications in recruitment strategy. Diversification of recruitment methods to include flyers and newspaper appeared to save considerable money as each of these methods was less expensive than TV. Diversification of recruitment methods also appeared to be valuable by reducing risk of being "trapped" using a single ineffective method and also by maximizing the chances of discovering more cost-effective methods. Interestingly, in this technology-driven era, it might seem unintuitive that a low-tech method of recruitment such as flyer placement would be more effective than TV or radio. UW-CTRI's results indicated that the use of flyers was not only the most time- and cost-effective recruitment method, but also broadened the ethnic distribution of the study sample.

Limitations

There are a few limitations on the description of this recruitment process, including:

• UW-CTRI's studies were designed to evaluate smoking cessation, not recruitment outcomes. As such, there is no control condition to clearly demonstrate how progressive modifications may or may not improve recruitment outcomes. The lack of a control group, for example, with an unchanging recruitment strategy, means also that researchers are unable to calculate the savings that might have occurred due to iterative methods.

• It is possible that the apparent success of recruitment results was due to an initial over-estimation of recruitment time and costs. There is, however, evidence to suggest the contrary. The cost of recruitment via television was roughly $260/enrollee, closer to the $300/enrollee found in the reference study.

• The statistics used in iterative analysis were rudimentary and lacked precision that might be gained through more sophisticated analyses. In most cases, decisions were made after at least several waves of data had been analyzed and major trends in data became noticeable. Larger, more expensive trials could benefit from employment of more sophisticated analyses to detect group differences.

• It is possible errors in judgment were made during the iterative assessment. For example, researchers persistently included TV advertising despite poor cost per enrollee.

• Recruitment response was provided as cause for gradual decline in newspaper viewings. This finding, however, could have also been due to other factors (i.e., competing newspaper advertisements).

• Madison, Wis., contains a relatively racially homogenous population. Studies within more substantially diverse populations might produce different results.

Conclusion

IIterative optimization of recruitment strategy is an intuitive approach to maximizing outcomes in human subjects recruitment. The method provided is low-tech, requires only rudimentary statistics, and, as such, should be accessible to most researchers and useful in either small or large studies. The process requires initial planning for data collection on recruitment outcomes and time for regular data reports on study recruitment. In studies conducted by UW-CTRI, this method of iterative optimization of recruitment strategy appears to have led to a decrease in cost and provided higher enrollment numbers than initially expected. It is likely that some studies operate in an optimal fashion without assessment, although many studies fail to reach a desired number of participants to obtain meaningful outcomes.¹⁴

Although some findings from this study may not generalize to all studies, the general principles of iterative optimization, including assessment of recruitment goals, collection, iterative analysis of recruitment data, and optimization of methods, should be useful to many. The hope is that researchers might benefit from this recruitment methodology and employ principles of this process to optimize outcomes as they see fit.

James M Davis, MD, Assistant Professor and Principal Investigator, University of Wisconsin Center for Tobacco Research and Intervention (UW-CTRI); Andrew J. Sandgren, Research Assistant, UW-CTRI; Alison R. Manley,*Study Coordinator, UW-CTRI, email: [email protected]; Stevens S. Smith, PhD,Associate Professor, Department of Medicine, UW-CTRI.

* To whom all correspondence should be addressed.

References

1. Campbell, M. K., Snowdon, C., Francis, D., Elbourne, D., McDonald, A. M., Knight, R., and Grant, A. (2007). Recruitment to randomized trials: strategies for trial enrollment and participation study. The STEPS study. Health Technology Assessment, 11(48), iii, ix-105.

2. Connett, J. E., Bjornson-Benson, W. M., and Daniels, K. (1993). Recruitment of participants in the Lung Health Study, II: Assessment of recruiting strategies. Controlled Clinical Trials, 14(2 Suppl), 38S-51S.

3. Epstein, S. (2008). The rise of "recruitmentology": clinical research, racial knowledge, and the politics of inclusion and difference. Social Studies of Science, 38(5), 801-832.

4. Harrison-Woolrych, M., Maggo, S., Tan, M., Savage, R., and Ashton, J. (2012). Cardiovascular events in patients taking varenicline: a case series from intensive postmarketing surveillance in New Zealand. Drug Safety, 35(1), 33-43.

5. Lovato, L. C., Hill, K., Hertert, S., Hunninghake, D. B., and Probstfield, J. L. (1997). Recruitment for controlled clinical trials: literature summary and annotated bibliography. Controlled Clinical Trials, 18(4), 328-352.

6. Harris, K. J., Ahluwalia, J. S., Catley, D., Okuyemi, K. S., Mayo, M. S., and Resnicow, K. (2003). Successful recruitment of minorities into clinical trials: The Kick It at Swope project. Nicotine & Tobacco Research, 5(4), 575-584.

7. McClure, J. B., Greene, S. M., Wiese, C., Johnson, K. E., Alexander, G., and Strecher, V. (2006). Interest in an online smoking cessation program and effective recruitment strategies: results from Project Quit. Journal of Medical Internet Research, 8(3), e14.

8. Okuyemi, K. S., Cox, L. S., Nollen, N. L., Snow, T. M., Kaur, H., Choi, W., and Ahluwalia, J. S. (2007). Baseline characteristics and recruitment strategies in a randomized clinical trial of African-American light smokers. American Journal of Health Promotion, 21(3), 183-191.

9. Graham, A. L., Lopez-Class, M., Mueller, N. T., Mota, G., and Mandelblatt, J. (2011). Efficiency and cost-effectiveness of recruitment methods for male Latino smokers. Health Education and Behavior, 38(3), 293-300.

10. Duffy, S. A., Scholten, R. L., and Karvonen-Gutierrez, C. A. (2010). The relation of tobacco use during hospitalization to post-discharge smoking cessation among US veterans. Preventive Medicine, 50(5-6), 285-287.

11. Frierson, G. M., Willioms, D. M., Dunsiger, S., Lewis, B. A., Whiteley, J. A., Albrecht, A. E., and Marcus, B. H. (2008). Recruitment of a racially and ethnically diverse sample into a physical activity efficacy trial. Clinical Trials, 5(5), 504-516.

12. Gren, L., Broski, K., Childs, J., Cordes, J., Engelhard, D., Gahagan, B., and Marcus, P. (2009). Recruitment methods employed in the Prostate, Lung, Colorectal, and Ovarian Cancer Screening Trial. Clinical Trials, 6(1), 52-59.

13. Heiat, A., Gross, C. P., and Krumholz, H. M. (2002). Representation of the elderly, women, and minorities in heart failure clinical trials. Archives of Internal Medicine, 162(15), 1682-1688.

14. Lovato, L. C., Hill, K., Hertert, S., Hunninghake, D. B., & Probstfield, J. L. (1997). Recruitment for controlled clinical trials: literature summary and annotated bibliography. Controlled Clinical Trials, 18(4), 328-352.

15. Pinsky, P. F., Ford, M., Gamito, E., Higgins, D., Jenkins, V., Lamerato, L., and Gohagan, J. K. (2008). Enrollment of racial and ethnic minorities in the Prostate, Lung, Colorectal and Ovarian Cancer Screening Trial. Journal of the National Medical Association, 100(3), 291-298.

16. Graham, A. L., Papandonatos, G. D., DePue, J. D., Pinto, B. M., Borrelli, B., Neighbors, C. J., and Abrams, D. B. (2008). Lifetime characteristics of participants and non-participants in a smoking cessation trial: implications for external validity and public health impact. Annals of Behavioral Medicine, 35(3), 295-307.

17. Shiffman, S., Dresler, C. M., Hajek, P., Gilburt, S. J., Targett, D. A., and Strahs, K. R. (2002). Efficacy of a nicotine lozenge for smoking cessation. Archives of Internal Medicine, 162(11), 1267-1276.

18. Taylor, C. B., Houston-Miller, N., Killen, J. D., & DeBusk, R. F. (1990). Smoking cessation after acute myocardial infarction: effects of a nurse-managed intervention. Annals of Internal Medicine, 113(2), 118-123.

19. Tonstad, S., Tonnesen, P., Hajek, P., Williams, K. E., Billing, C. B., and Reeves, K. R. (2006). Effect of maintenance therapy with varenicline on smoking cessation: a randomized controlled trial. Journal of the American Medical Association, 296(1), 64-71.

20. Webb, M. S. (2008). Focus groups as an intervention for low-income African American smokers to promote participation in subsequent intervention studies. Research in Nursing & Health, 31(2), 141-151.

21. Berman, B. A., Grosser, S. C., & Gritz, E. R. (1998). Recruitment to a school-based adult smoking-cessation program: do gender and race/ethnicity make a difference? Journal of Cancer Education, 13(4), 220-225.

22. Boyd, N. R., Sutton, C., Orleans, C. T., McClatchey, M. W., Bingler, R., Fleisher, L., and Ward, J. A. (1998). Quit Today! A targeted communications campaign to increase use of the cancer information service by African American smokers. Preventive Medicine, 27(5 Pt 2), S50-60.

23. Faseru, B., Cox, L. S., Bronars, C. A., Opole, I., Reed, G. A., Mayo, M.S., and Okuyemi, K. S. (2010). Design, recruitment, and retention of African-American smokers in a pharmacokinetic study. BMC Medical Research Methodology, 10, 6.

24. King, A. C., Cao, D., Southard, C. C., & Matthews, A. (2011). Racial differences in eligibility and enrollment in a smoking cessation clinical trial. Health Psychology, 30(1), 40-48.

25. Treweek, S., Mitchell, E., Pitkethly, M., Cook, J., Kjeldstrom, M., Taskila, T., and Jones, R. (2010). Strategies to improve recruitment to randomised controlled trials. Cochrane Database Systematic Reviews(1), MR000013.

26. Dauer, J. P., and Krueger, R.J. (1977). An iterative approach to goal programming. Operational Research Quarterly, 28(3), 671-681.

27. Heavlin, W. D., & Radovanovich, A. (2012, April 16-20, 2012). Risk-aware revenue maximization in display advertising. Paper presented at the WWW-2012: Session: Advertising on the Web.

28. Ignizio, J. P., and Romero, C. (2003). Goal Programming. Encyclopedia of Information Systems, 2, 489-500.

29. Heavlin, W. D., and Radovanovich, A. (April 16-20, 2012). Risk-aware revenue maximization in display advertising. Paper presented at the WWW-2012: Session: Advertising on the Web.