CRA Remediation Data Demonstrates Skills Improvement in Multiple Domains

In a previous article, we presented a global CRO’s data using an objective web-based monitoring simulation administered by CRA Assessments, LLC (CRAA). The analysis revealed that CRAs of all experience levels and in all regions are consistently underperforming in areas critical to maintaining subject safety, site compliance, study data integrity and quality. We also discussed some of the identified trends and possible reasons for these deficiencies.

In this article, we will demonstrate a methodology that resulted in improving CRA performance. We will look at another data sampling from a CRO that implemented a full CRA evaluation and remediation process (i.e. baseline simulation assessment, targeted training modules, and targeted re-evaluation simulations) to enhance CRA performance.

CRA Simulation and Remediation Efforts

The simulation method for evaluating and remediating CRA performance involves use of a web-based monitoring simulation and core-monitoring skills training modules. After completing a Baseline Simulation to determine areas of monitoring deficiency, the scores are grouped to identify deficiencies that can be addressed via the Targeted Training Modules, and an appropriate Threshold Score is determined by the Sponsor/CRO. When CRAs do not meet threshold score requirements in particular “Domains,” remediation plans for those individual CRAs are developed. Subsequently, Targeted Training Modules specific to clinical monitoring Domains are administered to address the CRA monitoring deficiencies identified by the baseline simulation. 

Once training is complete, stand-alone Follow-up Simulations are administered to serve as companion re-evaluations after each of the targeted training modules. This provides quantifiable insights into which Domains were improved versus those which remain a challenge for CRAs even after remediation. 

Data Sample Selection and Methodology

The data sample consisted of assessment and remediation efforts on 30 unique CRAs in North America for a global CRO interested in improving CRA skills-based monitoring performance in the following Targeted Training Modules: (1) ICF Process, (2) Protocol Review (3) IRB/IEC Submission on and Approval (4) Source Documentation, CRF, Source to CRF and (5) Potential Fraud/Scientific Misconduct and Delegation of Authority. Of the 30 CRAs, 3 had above-threshold scores on all Domains and did not undergo remediation training, 4 were assigned Domain-specific remediation training, but, did not complete their training, and 23 underwent remediation training and obtained follow-up simulation scores after their assigned Domain training. The Protocol Review domain was excluded from the remediation analysis because the sample size was too small; only two CRAs were assigned remediation training in that domain. This CRO selected any baseline score greater than 50% as the Threshold Score for adequate CRA performance in the Domains. The sample selection methodology consisted of ensuring that the data most appropriately reflected trends from the simulation, and computer data science algorithms from Python were used to select and visualize the data for analysis. In this analysis we used bell curves and bar charts to describe basic statistics, and leveraged means analyses to demonstrate changes of average scores from baseline to remediation. CRAA compensated Annex Clinical to conduct the data research portion of this analysis.

Results

Figure 1 shows that the Protocol Review Domain had the highest average score of 77%, whereas ICF Process (average of 60%), Source Documentation, CRF, Source to CRF Review (average of 57%), Potential Fraud, Scientific Misconduct, and Delegation of Authority (average of 54%), and IRB/IEC Submission and Approval (average of 44%) followed.

Figure 1: Bell Curves of CRA Scores in Varying Domains (N=30)

When evaluating the impact of remediation on baseline scores, only CRAs that scored 50% or lower (Threshold Score) in individual Domains were included in the following analysis, and the Protocol Review Domain was excluded due to its small sample size (N = 2). On average across all Domains, scores rose by 36%, bringing scores up from 33% at baseline to 69% after training and remediation.

Figure 2: Paired Categorical Plot of Mean Scores at Baseline and After Remediation

Figure 2 shows that remediation had the highest impact on changing the average scores of the ICF Process (a 47% increase), IRB/IEC Submission and Approval (a 43% increase), Potential Fraud, Scientific Misconduct, and Delegation of Authority (a 27% increase), and Source Documentation, CRF, Source to CRF Review (a 25% increase) followed.

Figure 3 shows that prior to remediation, an average of 53% of CRAs met threshold score requirements by having a score of over 50%. CRAs who were assigned remediation training had an average of 2.1 modules assigned.   After remediation, an additional average of 36% of CRAs met threshold score requirements, bringing 88% of CRAs to meet threshold score requirements. An average of 12% of CRAs did not meet threshold score requirements after remediation. 

Figure 3: Percentage of CRAs Meeting/Not Meeting Threshold Scores During Baseline and After Remediation (N = 26)

Discussion

Currently, few Sponsors/CROs objectively measure and remediate CRA performance. Albeit the sample size was small in this analysis, and the threshold score is variable from CRO/Sponsor company to company, this analysis provides preliminary evidence that standardized diagnostics and remediation helps Sponsors/CROs improve CRA performance. Knowing that CRAs that required remediation training were assigned an average of 2.1 modules, the fact that ICF issues and inadequate record keeping are in the top 5 common clinical investigator deficiencies ^[1], and failure to obtain FDA and/or IRB approval prior to study initiation is one of the most common Sponsor/CRO deficiencies according to FDA ^[1], Sponsors/CROs can seek to mitigate these risks through CRA assessment and remediation; CRAs who underwent remediation in this analysis saw discernable improvements in scores for domains pertaining to ICF process, IRB/IEC submission and approval, and modest, but, notable improvements in potential fraud/scientific misconduct and delegation of authority, and source documentation/source to EDC from baseline scores. “The CRAA evaluation and remediation process is well suited to provide the necessary exposure to real-world monitoring challenges that a CRA might face during a monitoring visit,” said Gustavo Poveda, COO of CRAA. “With the goal of reducing risk in clinical trials, use of a targeted, quantifiable approach to CRA evaluation and remediation allows Sponsors/CROs to more effectively and efficiently develop their CRAs,” added Gerald DeWolfe, CEO of CRAA.

Sponsors and CROs Can Assess and Improve CRA Monitoring Competency

While numerous monitoring training programs implemented by a variety of Sponsors/CROs may prepare a CRA for clinical trial monitoring, the variability and lack of predictability in CRA monitoring performance makes it a challenge for any sponsor/CRO to consistently and objectively evaluate and remediate CRA deficiencies. In our previous article, we demonstrated evidence of CRA deficiency in a variety of domains, and in this article, we presented preliminary evidence on how leveraging CRA diagnostics and implementing targeted remediation can improve CRA performance. FDA’s most common violations focus on the aforementioned domains, all of which can potentially be mitigated by proper diagnostics and remediation.

References:

[1] https://www.fda.gov/downloads/ScienceResearch/SpecialTopics/RunningClinicalTrials/UCM604510.pdf

Moe Alsumidaie, MBA, MSF is Chief Data Scientist at Annex Clinical, and Editorial Advisory Board member for and regular contributor to Applied Clinical Trials.