Does PI Certification Make a Difference?

The development of human therapeutics demands the successful completion of clinical trials. In the U.S., clinical trials must be conducted by qualified clinical principal investigators (PIs). The FDA does not specify minimum training for PIs in clinical research, but PIs should have appropriate training and experience to investigate an investigational product.¹ Most commonly, PIs are educated at the PhD or professional doctorate level. Although certifications like Certified Principal Investigator (CPI) exist, for example from the Association of Clinical Research Professionals (ACRP), PIs are not currently required to have them. The study outlined in this paper sought to determine if certified PIs produced higher quality clinical trial data than non-certified PIs.

Why certify?

In 2003, The National Institutes of Health (NIH) produced the NIH Roadmap, which called for the creation of innovative PI training programs to address barriers to translational development.² Over the next decade, educational systems developed programs with formal degrees, mentorships, and organized on-the-job training for clinical research students and practicing professionals transitioning to clinical research.

Although programs for PI training have grown and diversified, FDA audits have revealed persistent deficits in data quality.³ Since the mid-1990s, the same protocol deviations have been the top FDA audit findings.⁴ Protocol deviations, though significant in clinical trials, were actually found to be under-reported, which can threaten the internal validity of studies.⁵ Protocol deviations, especially those classified as major or critical deviations, can affect the usability of data, which impacts both cost and timely delivery of clinical studies, and subject safety.

Though not required for practice in clinical research, the only accredited certification for PIs is the certified principal investigator (CPI). Two studies have been conducted on the CPI. Although both studies have significant limitations, each study found a positive difference in certified PIs over non-certified PIs with respect to PI outcomes, protocol deviations,⁷ and FDA audit outcomes.⁸

Continuing education (CE) for PIs may be part of the solution for resolving these persistent deficits. CE has been shown to improve professional practice, including medical practice.⁶ Since CE is generally required to maintain certification, requiring PIs to become certified would logically require a level of CE that could improve practice. Improved practice could improve data quality by reducing key quality indicators such as major and critical protocol deviations.

Because persistent deficits in the quality of data produced by PIs exists, and initial training and CE for PIs are not standardized, this study sought to determine if PI certification, via the CPI, could positively impact the quality of clinical trial data as determined by the number of major and critical protocol deviations produced by PIs participating in the study.

The study methods

The research question, “Is there a difference in the mean protocol deviation before and after clinical PI certification?”, was tested using a quasi-experimental, single-group interrupted time-series design.⁹ The mean protocol deviation was calculated for each study participant from protocol deviations produced retrospectively, from May 1, 2014 to April 30, 2016. This mean protocol deviation was compared to the mean protocol deviation for each study participant prospectively from Oct. 11, 2016 to March 22, 2017.

U.S. physicians who conducted clinical trials with a large contract research organization (CRO) since May 1, 2014 and who planned to conduct clinical trials with the same CRO through March 22, 2017 was the population of interest. A sample of participants from the population was selected based on certain inclusion and exclusion criteria.

Key inclusion criteria were the following:

• Participants required to be a U.S. MD or Doctor of Osteopathy (DO).

• Participants required to be eligible for the Fall 2016 certified physician investigator (CPI) exam cycle.

Key exclusion criteria were the following:

• Participants who were currently enrolled, intended to enroll in, or had previously completed a clinical research education program that met the ACRP standards as a substitute for one year of the two years of required employment experience as a principal investigator (PI) to be eligible for the CPI exam.

• Participants with a research-related dual degree such as MD/DO-PhD, MD/DO-MPH or MD/DO-MSCR.

• Participants who were currently certified, had previously been certified, or had previously failed the CPI exam.

Prescreening using information in the CRO and ACRP’s databases was performed to narrow the number of participants contacted for screening. It was projected that 500 participants would need to be prescreened to identify 40 participants. 40 participants were expected to be needed to obtain 20 evaluable participants to account for ACRP’s historic exam withdrawal rate of 10% and historic exam failure rate of 45%. To account for the possibility of a non-normal distribution of data and to show a significant effect, if one existed, a large enough sample size, 20 participants, was chosen to allow for parametric testing.¹⁰

CRO training modules for GCP and access to CPI practice exams were provided to participants. Although this material was offered, there was no way to confirm the amount of preparation each participant made for the exam.

CPI is the only accredited certification for clinical PIs and is accredited by the National Commission for Certifying Agencies (NCCPA). The CPI is administered by the Academy of Clinical Research Professionals and is awarded by the Association for Clinical Research Professionals. The CPI exam is administered twice annually, once in the spring and once in the fall. To be eligible for the study, participants had to meet the criteria outlined by the Association for Clinical Research Professionals (ACRP)¹¹ for the Fall 2016 testing cycle.

Study variables

The dependent variable was the mean protocol deviations calculated from May 1, 2014 to April 30, 2016 and from Oct. 11, 2016 to March 22, 2017 for each study participant. The independent variable was PI before certification (0) compared to PI after certification (1).

Protocol deviations produced and clinical trial subjects enrolled by the study participants were downloaded from the CRO’s database. The major and critical protocol deviations were added together and divided by the number of clinical trial subjects recruited by each PI and active during the periods of interest. This created the dependent variable, mean protocol deviation for each participant. The CRO’s definitions for major and critical protocol deviations were used for the study:

• Major Protocol Deviation: Deviation from protocol that impacts scientific, ethical, regulatory, or business integrity and which if left unattended could become critical.

• Critical Protocol Deviation: Deviation from protocol that impacts scientific, ethical, regulatory, or business integrity and could invalidate acceptability of a study to a manufacturing sponsor or regulatory body or invoke regulatory action.

Analysis

Data analysis was performed using SAS. Demographic data collection was used to describe participant characteristics, including medical specialty and medical degree type.

Descriptive statistics included means, standard deviations, and ranges for the following:

• Years of medical practice experience post completion of residency.

• Years of clinical trial research experience in the PI role.

• Clinical trial subjects enrolled by participants, major and critical protocol deviations produced by participants, and mean protocol deviations produced by participants for each period of interest.

Because the standard deviation of the population of PIs that fit the inclusion and exclusion criteria of the study was unknown, paired-samples t-tests were used to draw conclusions about this population. Paired-samples t-tests were used to compare the mean protocol deviations of the study participants from May 1, 2014 to April 30, 2016 and those from Oct. 11, 2016 to March 22, 2017. This provided a before-and-after analysis of the treatment intervention using each participant as his own control. Effect size was calculated using Hedges’ g calculation for t-test.

Results

The breakdown of the study population from prescreening to evaluable participants is reflected in the study procedure (Figure 1). The response rate from screening was 2.4%. The exam withdrawal rate was 25%, which was higher than ACRP’s historic exam withdrawal rate of 10%. The exam pass rate was 93.3%, which was substantially higher than ACRP’s historic pass rate of 55%. Twelve participants, 1.2% of the study population, were evaluable.

All participants held the degree of Doctor of Medicine, MD. The predominant medical specialties were gastroenterology and rheumatology, representing four participants each. On analysis of Table 1, all participants claimed over 10 years of medical practice experience, and 75% of participants claimed over 10 years of clinical research practice experience.

As shown in Table 2, research experience was found to be a statistically significant predictor of mean protocol deviations; however, this effect was a neutral one in that it was present both before and after certification. Table 3 presents the descriptive statistics for the dependent variable, mean protocol deviations. 

Table 4 reflects the results of the paired–samples t-test, which showed a significant difference in the mean protocol deviation of PIs before taking the CPI exam (M = .5792, SD = .6751) and after passing the CPI exam (M = .1250, SD = .2560); t (2.3325), p < .05. This study’s results suggest that a PI’s mean protocol deviation drops an average of 0.4542 major and critical protocol deviations per enrolled clinical trial subject after passing the CPI exam and becoming certified. Using Hedges’ g for t-tests, the effect size was found to be large at 0.8588 and demonstrated a strong connection between the variables.

Discussion

The literature shows that continuing medical education (CME) improves medical practice⁶ and that CME is the accepted basis for medical re-licensure and recertification.¹² This study suggests that there is an impact to PI data quality after initial certification, but does not demonstrate that this is related to continuing education. Additional data collection through the time of recertification would be required to demonstrate this. However, it is reasonable to assume that like medical practice, clinical research practice would show a similar pattern of practice improvement.

CPI certification could offer a reasonable alternative to PIs who cannot afford the time away from medical practice or the financial burden to complete a formal, degreed, clinical research program to demonstrate competency in this specialty. Although additional certification may be perceived as a burden by some, most of the PIs contacted for this study expressed a need for standardization and certification. Further, they believe the evidence that physicians’ knowledge and skill decrease over time and that once-in-a-lifetime certification or training could not ensure competency throughout a career.¹³ In a study of physician perceptions of the maintenance of certification process, it was noted that physicians accept the concept of specialty recertification as a good idea.¹³ This is consistent with the anecdotal responses from the population sampled in this study.

Some PIs who responded positively to being solicited for the study also saw certification as a way to differentiate themselves from their peers.  

Limitations

This study had several limitations. All participants had a working relationship with at least one CRO and may, therefore, be more skilled clinical researchers than PIs without this affiliation. Additionally, PIs were selected from trials running in several therapeutic areas. The results cannot be generalized outside of the specialties studied and should not even be generalized for those specialties represented secondary to the small sample size from each specialty.

The study sample size was small. Secondary to the researcher’s position within the CRO providing data, it was believed to be unethical to heavily solicit participants to participate in the trial. Had this not been a concern, more aggressive recruitment techniques could have been employed and would have likely resulted in a greater sample size for analysis.

The study was a quasi-experimental design that allowed participants to self-select to participate when solicited. Participants could, therefore, be predisposed to be more motivated and higher performing than the general PI population in the CRO’s databases. Additionally, participants might have improved their protocol compliance for no other reason than their data was being monitored. Both instances could favorably influence the results.

Though the effect size between means evaluated proved to be large, the sample size was less than desired, and the time for data collection-five months-could have been longer. A greater sample size and extended study timeline could have affected the study results in either direction.

Suggestions for future research

Future research should include a larger, experimental design that further limits self-selection bias. Future studies should involve participants affiliated with CROs and those unaffiliated with CROs. Participants should be stratified by particular clinical trials and by years of PI practice experience so that trial complexity and experience are not confounding factors. Participants in future trials should be followed long-term to determine if the apparent short-term effect on mean protocol deviations is sustained. Recommended long-term follow-up could include analysis at one year, two years (first recertification cycle) and four years (second recertification cycle).

Conclusion

This study suggests that certification, with its continuing education maintenance requirement, may improve clinical research practice. Certification of PIs should be considered as a means of expanding the number of PIs and improving the quality of data produced by PIs. If further research in this area validates these initial results, both industry and regulatory bodies should consider certification as a requirement to perform the duties of the PI.

Kathryn Rena Hodges, PhD, MSCR, PA-C, UNCPN, Knightdale Family Medicine, and Manager, Clinical Trials Operations, Duke Clinical Research Institute; email: kathryn.hodges@unchealth.unc.edu; Duane Akroyd, PhD, Professor, North Carolina State University; email: duane_akroyd@ncsu.edu

References

1. United States Food and Drug Administration. Code of Federal Regulations Title 21 2015. https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfCFR/CFRSearch.cfm?CFRPart=312 Accessed June 13, 2015.

2. Zerhouni E. The NIH Roadmap. Science. 2003;302(5642):63-64.

3. Association of Clinical Research Professionals Annual Conference. FDA Presentation 2010. http://www.acrpnet.org/MainMenuCategory/Education/OnlineConferenceLibrary.aspx. Accessed June 15, 2015.

4. Getz K, Campo R. Drug Development Study Designs Have Reached the Danger Zone. Expert Rev Clin Pharmacol. 2013;6(6):589-591.

5. Sweetman EA, Doig GS. Failure to Report Protocol Violations in Clinical Trials: A Threat to Internal Validity. Trials. 2011;12(214):1-8.

6. Davis D, O’Brien MAT, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vassey A. Impact of Formal Continuing Medical Education: Do Conferences, Workshops, Rounds, and Other Traditional Continuing Education Activities Change Physician Behavior or Health Care Outcomes? JAMA. 1999;282(9): 867-874.

7. Haeusler JC. Certification in good clinical practice and clinical trial quality: A retrospective analysis of protocol adherence in four multicenter trials in the USA. Clin Res Regul Affairs. 2009;26(1-2):20-23.

8. Vulcano DM. (2012). CPI Certification as Predictor of Clinical Investigators’ Regulatory Compliance. Drug Inf J. 2012;46(1):84-87.

9. Creswell JW. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Thousand Oaks, CA: SAGE Publications; 2014.

10. Harris MB. Basic Statistics for Behavioral Science Research. Needham Heights, MA: Simon & Schuster; 1995.

11. Association of Clinical Research Professionals. PI Certification 2015. http://www.acrpnet.org/MainMenuCategory/Certification/PICertification.aspx. Accessed June 27, 2015.

12. Cervero RM. Continuing Professional Education in Transition, 1981-2000. Int J Lifelong Educ. 2001;20(1-2):16-30.

13. Cook DA, Holmboe ES, Sorenson KJ, Berger RA, Wilkinson JM. Getting Maintenance Certification to Work: A Grounded Theory Study of Physicians’ Perceptions. JAMA. 2015;175(1):35-42.