Audit Preparation Class: Does It Really Help CRAs?

The Clinical Research Associate (CRA) Discipline of the Children's Oncology Group (COG) is comprised of CRAs from approximately 235 institutions throughout North America, Switzerland, Australia, and New Zealand. With approximately 95% of children and adolescents in North America seen at COG institutions, and many enrolled in clinical trials, it is imperative to protect subjects through regulatory compliance. Audits are recognized as a measure of the regulatory compliance, data, and pharmacy processes at each site. At the semi-annual COG meetings, there is an audit preparation class offered to institutional Principal Investigators (PI) and CRAs.

Photography: Comstock

Purpose

The purpose of this study was to determine if attendance at the audit preparation class would help the institutional CRA prepare for an audit, and if this preparation would impact the audit result.

Background and significance

Clinical research is a cornerstone in the advancement of a cure for cancer. An audit is an integral part of this process and ensures the validity of research data from which conclusions are drawn. Here we refer to an audit as the comparison of data abstracted for analysis with that recorded in the primary medical record and with the research records of the investigative site. Such activities have revealed scientific fraud^1-3 that would otherwise have gone undetected.

In order to maintain the integrity of the audit process, auditors are necessarily independent of the sites at which the data are generated. Thus, preparing for an audit, including the preparation of institutional records that verify reported data, is generally the responsibility of the institutional CRA. This preparation is crucial, since questions raised on audit often require a lengthy resolution process that can involve sanctions against the site.

At COG, the CRA Discipline undertook an educational program for audit preparation. We herein report the assessment of the intervention.

Education courses

Much research has been done about the benefit of education, application, and outcome. Tarenetal.⁴ recorded the efficacy of an integrated undergraduate medical curriculum to increase the clinical skills of medical students, and found that the students did improve their clinical practice skills (in nutrition) through participation in an integrated nutrition curriculum. Similarly, Kellmer-Langan⁵ related that nursing staff development programs increased knowledge and improved skills of nurses. Chiplin⁶ reported that continuing education courses improved the practice of physiotherapists.

CRA Survey Questions

A search of the Cochrane Database of Systematic Reviews⁷ provided ample evidence to support the importance of educational meetings and printed educational materials. They are sighted as the two most common types of continuing education for health professionals, with the aim of improving professional practice. The selection criteria included randomized trials or well-designed, quasi-experimental studies examining the effect of continuing education meetings (including lectures, workshops, and courses) on the clinical practice of health professionals or health care outcomes. The conclusion was that interactive workshops could result in moderately large changes in professional practice. Didactic sessions alone were unlikely to change professional practice.

A further search of the Cochrane database⁸ provided support for access to printed educational materials in improving the behavior of health care professionals. The selection criteria included randomized trials, interrupted time series analyses, and nonequivalent group designs with prepost measures of interventions comparing both: printed educational materials versus a nonintervention control and printed educational materials plus additional implementation strategies versus printed educational materials alone. The participants were any health care professionals provided with printed educational materials aimed at improving their practice. The conclusions suggested that the effects of printed educational materials compared with no active intervention appeared small and of uncertain clinical significance. The additional impact of more active interventions produced mixed results.

Clearly, there is evidence to support the hypothesis that education courses improve practices. This is especially important in the field of research, where audit performance dictates the status of institutions with respect to human subject protection, regulatory compliance, and the continued ability to enroll pediatric oncology patients into research studies. Loss of the ability to enter pediatric oncology patients into research studies would compromise their care.⁹

We propose that should this study show improved audit performance for institutions whose CRA had attended the audit preparation course, then the audit preparation course be made mandatory for lead institutional CRAs.

Hypothesis

The audit preparation course improves the institutional CRA's audit rating and audit procedures.

Subjects

The COG CRA Discipline committee and COG Regulatory Compliance Office conducts a half-day audit preparation training course once every six months during the COG semi-annual meeting. Participation in this course, while voluntary, is strongly encouraged by the leadership of the COG CRA Discipline committee and Regulatory Compliance Office. As part of an ongoing evaluation of this effort, we distributed a questionnaire to all MD and CRA or nurse auditors and local lead CRAs scheduled to be involved in the audit process in the 12-month period from January to December 2003.

Method

This was a prospective study to determine the outcome of CRA participation in the audit preparation class, offered as part of the CRA continuing education program.

Auditor Survey Questions

Based on the National Cancer Institute's (NCI) Clinical Trials Monitoring Branch guidelines, COG administrative procedures require an on-site audit of each of the approximately 235 participating hospitals at least every three years. The quality assurance and audit personnel at the COG Group Operations Center, who schedule all audits, identified institutions that were to be audited in the 12-month period ending December 2003. A package of information containing questionnaires and instructions was mailed to the auditor prior to the audit and was brought to the site of the audit for distribution. Verbal consent was obtained from the MD and CRA or nurse auditors as well as the local lead CRA. The instructions were to complete the questionnaire as soon as possible after the audit.

Using a Likert scale, the MD and CRA or nurse auditors rated their opinion of various portions of the site's audit preparation (organization of materials, quality of prepreparation, knowledge of procedure, outcome of audit, how feedback was received). The local lead CRA rated their impressions of the audit separately (see Table 1). The questions did not probe the outcome of the audit. These evaluations were obtained from the standardized NCI report of audit results. In addition to questions regarding the audit process and outcome, we collected data on factors believed to be related to the ability of the institutional team to prepare for an audit (see Table 2). (A complete questionnaire is available from COG upon request.)

We elected to analyze the data after using the questionnaire for one year. The analysis time point was not chosen to attain specified power characteristics for selected comparisons, but was done as part of a compliance review of participation by institutional investigators and auditors in the questionnaire process.

The relationship between investigator experience and participation in the audit preparation course and evaluation of the site's preparation for the audit was assessed by categorical data methods.¹⁰ P-values less than 0.05 (2 sided) were considered significant findings for this exploratory analysis. For most analyses, the Likert scale was grouped to form two categories: Acceptable—good, very good, or excellent; and Not Acceptable—fair or poor.

Results

Compliance with this survey was good, with 78% of MD and CRA or nurse auditors returning their surveys at the end of the audit.

None of the measures of preparedness for the audit were related significantly to the number of years the lead CRA was involved in COG research. The value of participation in the semiannual COG investigators' meeting, however, was reflected in several major categories of audit preparation. Overall, preparation for an audit was considered acceptable for 74% of the audits where the institutional CRA had attended at least three COG meetings compared with 42%–64% when the CRA had attended two or fewer meetings. Similar results were observed with other measures of preparedness.

This trend was observed with other variables compared to the number of COG meetings attended throughout the analysis. The variable of "pharmacy appointment and knowledge" showed 80% of CRAs with meeting attendance of 3+ received a rating of good/very good/excellent, while those attending less than two meetings received that rating only 51% to 64% of the time. With the variable "knowledge of the audit process," 77% received a rating of good/very good/excellent with meeting attendance of 3+, while those who attended less than two meetings received this rating only 42% to 52% of the time.

When analyzing variables and comparing them to whether or not the lead CRA attended the audit preparation class, similar findings were seen. Considering the variable of "pharmacy appointment and knowledge," 83% of CRAs received a rating of good/very good/excellent when attending the audit preparation class, while only 50% to 65% of nonattendees received this rating. Likewise, in "knowledge of the audit process," 77% received a good/very good/excellent rating compared to 40% to 61% when CRAs did not attend the audit preparation class. When comparing the "overall preparation for the audit" to whether or not the lead CRA attended the audit preparation class, those who did attend scored good/very good/excellent 77% of the time compared to 40% to 65% of those who did not attend the class.

Not surprisingly, we found that "knowledge of the audit process" was linked to whether or not this was the CRA's first audit. Those who had participated in more than one audit received a good/very good/excellent rating in 79% of cases compared to 47% of those for whom this was the first audit.

Another interesting finding was that when other staff attended the audit preparation course there was a higher percentage of good/very good/excellent ratings in all of the above categories.

Discussion

The goal of this survey was to determine if participation in the CRA audit preparation class at the semi-annual COG meeting improved the institutional CRAs' audit outcome. This outcome was measured by the categories in Table 1. While the findings were not statistically significant when analyzed by Fisher's Exact Test, there is clearly a trend showing that improved audit preparation and outcome is evident for CRAs who attend the audit preparation class, CRAs who attend more than three COG meetings, and when other staff members attend the audit preparation class.

It is interesting that attendance at COG meetings is significant for audit outcome. Depending on institutional enrollment for the proceeding three years, COG institutions are allocated monies from the organization to provide travel expenses for the PI, lead CRA, and other staff members to attend the COG semi-annual meetings. At each COG meeting there is approximately eight hours of new CRA training/education and 12 hours of continuing education for more experienced CRAs. This training is produced by the CRA discipline under the guidance of the CRA steering committee in collaboration with the COG's Statistical Data Center and the Group Operations Center.

It will be important to maintain the current funding of travel for institutional CRAs to ensure continued success with institutional audits.

Acknowledgements

The authors would like to thank all the auditors and institutional CRAs who participated in this survey, the COG Regulatory Staff for their assistance in mailing and compiling the data from this survey, and Zhengjia (Nelson) Chen for analyzing all the data.

Kim Nagel,* RN, CCRP, is lead CRA at McMaster University, 1200 Main Street W., Room 3E24, Hamilton, ON L8N325 Canada, (905) 521-2100 ext. 73228, fax (905) 521-5074, email: nagel@hhsc.ca. Susan Devine, CCRP, is CTSU manager at the Hospital for Sick Children, Toronto, Canada. Mark Krailo, PhD, and Susan Caso, manager of quality assurance and audit, are both with the Children's Oncology Group–Operation's Center, Arcadia, CA. Lisa Benson, CCRP, is manager of research administration–finance at the Connecticut Children's Medical Center, Hartford CT.

*To whom all correspondence should be addressed.

References

1. W.R. Bezwoda, L. Seymour, R.D. Dansey, "High-Dose-Chemotherapy with Hematopoietic Rescue as Primary Treatment for Meta static Breast Cancer. A Randomized Trial," Journal of Clinical Oncology, 13 (10) 2483–2489 (1995).

2. G.P. Canellos editorial, "The Policing of Clinical Trials," Journal of Clinical Oncology, 18 (12) 2353 (2000).

3. R.B. Weiss, G.G. Gill, C.A. Hudis, "An On-Site Audit of the South African Trial of High-Dose Chemotherapy for Metastatic Breast Cancer and Associated Publications," Journal of Clinical Oncology, 19 (11) 2771–2777 (2000).

4. D.L. Taren, C.A. Thomson, N.A. Koff et al., "Effect of an Integrated Nutrition Curriculum on Medical Education, Student Clinical Performance, and Student Perception of Medical-Nutrition Training," American Journal of Clinical Nutrition, 73 (6) 1107–1112 (2001).

5. D.M. Kellmer-Langan, C. Hunter, J.P. Nottingham, "Knowledge Retention and Clinical Application after Continuing Education," Journal of Nursing Staff Development, 8 (1) 5–10 (1992).

6. J.R. Chiplin and S.F. Bassett, "Continuing Education Programmes: Their Effectiveness in the Maintenance of Physiotherapists' Professional Competency," New Zealand Journal of Physiotherapy, 30 (3) 24–31 (2002).

7. N. Freemantle, E.L. Harvey, F. Wolf et al., Printed Educational Materials: EBM Reviews—Cochrane Database of Systematic Reviews, Cochrane Effective Practice and Organization of Care Group, Cochrane Database of Systematic Reviews, 3 (2005).

8. M.A. Thomson O'Brien, N. Freemantle, A.D. Oxman, Continuing Education Meetings and Workshops: EBM Reviews—Cochrane Database of Systematic Reviews, Cochrane Effective Practice and Organisation of Care Group, Cochrane Database of Systematic Reviews (3, 2005).

9. S.B. Murphy, "The National Impact of Clinical Cooperative Group Trials for Pediatric Cancer," Medical Pediatric Oncology, 24 (5) 279–280 (1995).

10. Y.M.M. Bishop, S.E. Feineberg, P.M.W. Holland, Discrete Multivariate Analysis (MIT Press, Cambridge, MA, 1975).