Critical Appraisal of Clinical Quality Assurance Based on Benchmarking Results: Past - Present - Future

All stakeholders, from regulatory authorities¹ and pharmaceutical industries,² unanimously request effective and efficient quality management systems (QMS). However, inspection results³ as well as recent quality concerns⁴ demonstrate the need for continuous improvement.

Therefore, development of a strategic organizational structure focusing on the subject of quality management has become a top priority in many pharmaceutical companies. Since clinical quality assurance (CQA) plays an integral role within the QMS, this makes it all the more important that up-to-date benchmarking information on CQA systems is evaluated.

Our previous benchmarking results⁵ provided an overview of CQA best practices within the pharmaceutical industry 10 years ago. A recently conducted comprehensive literature review did not reveal relevant updates on the subject. Therefore, a follow-up benchmarking survey was conducted to assess current practices. The survey among global pharmaceutical companies of different sizes focused on the organization, resources, strategies, and activities of CQA.

In this article, we critically review the changes over the past 10 years and provide recommendations on how to shape the future.

Benchmarking information regarding organizational structure, capacities, and strategies is difficult to obtain in a highly competitive environment. In 2004, we managed to receive information from 19 global companies. In our current survey, we received feedback from 12 global pharma companies, which also reflects mergers that took place in the past 10 years.

Organization and resources of CQA

Organizational structures of the participating companies differ substantially. Compared to the results in 2004, there is a shift of CQA reporting to a higher management level. Also, the current results are showing no instances where CQA reports into the clinical development team. This finding perfectly reflects the necessity of independent CQA units in the spirit of ICH-GCP⁶ (refer to section 2.13, 5.1, and 5.19). This is also in alignment with guidelines such as International Auditors (IIA) Standards,⁷ Guideline on Good Pharmacovigilance Practices (GVP) – Module IV, ICH Q10 (refer to section 2.1.4),⁸ where a harmonized and independent “quality unit” is requested.

Ten years ago, the results showed a clear relationship between R&D budget and CQA capacity. This reflected the obvious relationship between R&D activities and the required quality management (QM) support. We did not find this relationship replicated in the present results. Whereas, in 2004 most elements of QM support were provided by centralized CQA, today this is much more heterogeneous. In some companies, quality and compliance functions are implemented in a decentralized model within business units with less increase of CQA staff. In addition, the much higher level of outsourcing of clinical research activities, either including QM support (fully outsourced models) or keeping QM support in-house, contribute to the heterogeneity seen in the current survey.

Strategy and activities of CQA

In 2004, CQA capacity was spent relatively homogenously; besides the core activity of auditing (~70%) consultancy, SOPs, and training accounted for about 5% to 10% each. Today, auditing, including audit conduct and management of contracted audits, still reflects the highest share of CQA capacity, but has decreased to about 50%. Whereas, capacity spent for consultancy and training remained the same, it increased from about 5% to 10% for SOP management.

Today, CQA departments take over a broader variety of tasks. In particular, inspection management; non-compliance investigations, including root cause analysis; and involvement in process improvement projects are now typical activities of CQA and account for 5% to 10% each. This matches the decrease of capacity spent for auditing. Another new activity is vendor oversight other than audits (e.g., vendor selection) by CQA, which is performed by more than half of the pharmaceutical companies surveyed.

Figure 1: Distribution of CQA activities in 2014 displayed by minimum and maximum values, upper and lower quartile, and median (%). Count [n] reflects the number of companies (total n =12) performing a certain activity.

The CQA of one company performs quality control for other departments. In the view of the authors, this is not in line with the independence of CQA and furthermore undermines the accountability of the individual departments in regards to Quality Control steps.

Outsourcing of CQA activities

Outsourcing of certain tasks is very common in pharmaceutical companies. The number of organizations outsourcing CQA activities increased from 63% in 2004 to 75% in 2014.

Investigator site audits are outsourced most frequently. The in-house capacity necessary for handling and supervising outsourced audits is still about 30%. The outsourced audit activities are budgeted for by CQA in about 75% of the companies. All this did not change in the past 10 years.

Although outsourcing of vendor and system audits increased, it is still a lesscommon practice. Because vendor audits are primarily performed by internal auditors, this helps assure the company’s own quality standards are fostered and knowledge about the vendors is maintained within the company.

The majority of companies outsource audits either to specialized audit CROs or individual independent auditors. In this benchmarking survey, 25% of the companies stated that they outsource audit activities to the CRO that is also in charge of conducting/monitoring the respective trial. This increased from about 5% in 2004 and might be a consequence of full package outsourcing. Due to potential conflict of interest, the authors strongly recommend to separate out CQA outsourcing that better reflects ICH GCP 5.19.2: “The sponsor should appoint individuals who are independent of the clinical trials/systems to conduct audits.”

In the current survey, we also explored outsourcing of activities other than auditing. Except for SOP management and vendor oversight, all other activities are outsourced to some extent, most prominently internal training and training/education of CQA staff. This provides the advantage of getting input from third parties (e.g., external trainers).

In contrast, outsourcing of SOP management and vendor oversight is not commonly done, as it requires that external consultants are contracted on a long-term basis. This is conducted as such so the consultants can acquire sufficient knowledge of the company’s processes and responsibilities in order to provide beneficial support.

Audit strategies

Today, nearly all companies (with one exception) establish an annual audit program. Out of these, the great majority apply a risk-based approach, most of them using a formal scoring system for risk factors.

A variety of risk factors are taken into consideration for establishing the audit program and the most common risk factors named are as follows:

• Trial audits
• Trial type/importance (pivotal, dose-finding, proof-of-concept, supporting safety, bio-equivalence, post-approval, etc.)

• Special populations /vulnerable subjects

• Trial/protocol complexity

• Novelty of design, indication, or technology

• Registration intent (e.g., first registration [NCE] vs. line extension vs. publication)

• Previous compliance history (monitoring / site management)

• System audits
• Product portfolio (importance/properties of projects)

• Regulatory environment

• Workload/volume of activities (e.g., number of PV cases or Phase I trials)

• Prior compliance record (audit and inspection history)

• Risk levels of trials managed

Based on the audit strategy, audit coverage for trials, systems, and vendors (CROs) is determined. While risk factors already played a role in 2004, today the use of a risk-based strategy is much more pronounced and formalized with explicitly defined criteria, scoring rules, and associated tools.

Regarding trial audits, about one-third of the companies focus their audit activity on pivotal trials; about one-third state they have all trials from Phase I to IV in scope; the remaining companies determine audit coverage according to the risk level assigned to the trials. One company does not target trials, instead focusing on systems and, therefore, certain studies within the scope of their system audits. Compared to 2004, the sample of audited sites per trial selected for audits has decreased: typically, it now lies in the range of 5% to 10%, whereas 10 years ago, samples were in the range of 10% to 20%.

Regarding document audits, about half of the companies do not perform them at all; the other half select clinically important documents (protocol, IB, TMF, trial report, submission dossier)-mostly on a sample basis. There is a clear decrease in selecting documents for audits in comparison to the practices in 2004.

For system and vendor audits, the approach is relatively homogeneous across companies. Risk levels assigned to systems/vendors guide the selection for audit and also determine the audit cycle times, which range from two to five years.

The following is a typical response regarding system audits: “Selected based on risk. Various risk rating tools employed. Cycle time determined by risk, typically every three to five years.”

Another response regarding vendor audits: “Volume and importance of work. Time since last audit with identified issues. Preferred providers are audited every two years.”

In contrast to 2004, risk-based approaches are now widely applied to trial, vendor, and system audits.

Audit performance

The number of trial, system, and vendor audits performed per year and audit type varies considerably across companies (e.g., for investigator site audits the number varies from nine to 600). The number of investigator site audits is nearly perfectly correlated to the number of trials (r=0.99).

The majority of audits still focus on trials (investigator sites and trial documents) but their proportion decreased in favor of system and vendor audits. This clear trend is in line with the reduction of sample size for investigator site audits reported above.

Figure 2: Distribution of trial, vendor and system audits compared 2004 (n = 3310, total audits performed across 19 companies) and 2014 (n = 2592 total audits performed in 12 companies)

The shift from trial to vendor and system audits surely reflects the increase of outsourcing and the use of additional technologies. In our view, however, the main driver of this change is the paradigm shift towards a holistic QM system with focus on processes and systems. This interpretation is also supported by the regulatory agencies’ views (e.g., the Report on the Pilot EMA-FDA GCP Initiative’s conclusion: “To focus the joint inspections on sponsors and CROs instead of investigator sites in order to work towards developing a truly harmonized quality-systems approach to sponsor/CRO inspections.”⁹

The trend seen is very likely to continue to achieve a healthy balance between focusing the audit strategy on processes and systems supported by trial audits (ISO9001:2008 – Guidance in the concept and use of the process approach for management systems).¹⁰

In addition to audits, we asked for the number of pre-inspection visits performed per year. Again, this varies considerably across companies (range 0 and 20 visits).

Resource assumptions for the different audit types have been collected and analyzed. In general, more capacity is spent on system and vendor than on trial audits. This is also reflected in the median number of man-days spent for an audit:

• Investigator site audits: 12 man-days

• Vendor audits: 18 man-days

• System audits: 28 man-days

Compared to 2004, the capacity spent for an investigator site audit increased by about 50% (2004: 8 man-days, 2014: 12 man-days).

In summary, compared to practices in place 10 years ago:

• Risk-based approaches are currently more formalized and applied to all types of audits

• More capacity is allocated to individual audits, but fewer sites tend to be audited per trial

• Fewer document audits tend to be performed; it is rather the exception than the rule

• Finally, the proportion of system and vendor audits clearly increased reflecting a possible paradigm shift in the audit approach

Impact and reporting of audit findings

The process of audit reporting is quite standardized amongst companies and did not significantly change compared to 2004:

• Evaluation of significance of audit findings (e.g., critical, major, minor/other)

• Content-related categorization (e.g., study medication, monitoring, etc.)

• QC-steps for reporting of audit findings (e.g., peer review of audit reports)

• Expedited reporting process for critical findings (e.g., immediately to up to three business days)

However, a process for expedited reporting and classification of audit findings has become more common, which is a positive development towards increasing quality and objectivity.

Distribution of audit reports to various management levels supports the necessary management attention. Some companies also provide the full audit report to external auditees such as third-party vendors, but it seems more common to provide either a redacted version or information on the respective findings that relate to external auditees.

Systematic analysis and follow-up of audit findings

The majority of companies perform a systematic and periodic analysis of audit findings across audits (e.g., per trial or per project, quarterly and/or annually).

In comparison to 2004, the percentage of such analyses has further increased, which is an important step to enhance the impact of audits away from the corrections of single findings to a higher level of systematic improvement and/or prevention of re-occurrence.

Nowadays, the majority of companies involve senior management regarding audit trends to trigger implementation of quality improvement activities. This should be encouraged by all companies since management support is an important prerequisite for quality improvements. This quality management review is also required by ISO9001 or ICH Q9/Q10.

Corrective action/preventive action (CAPA) and follow-up

Today, the term "CAPA" is well established, whereas in 2004, the common term in the GCP-regulated area was "corrective action" only. Authorities are now expecting robust CAPA systems moving away from correcting single findings in isolation to preventive actions guided by a thorough root cause analysis. In case of critical and major findings, today CQA is always involved in establishing the CAPA plan. Tracking CAPAs resulting from audits is more common than checking, and the degree of CQA involvement is related to criticality. A check of CAPA implementation resulting from critical findings is done in about 90% of the companies, in 30% in case of major findings, but not in case of minor findings. Beyond this, for further activities, like review approval, etc., our data show a broad variety of CQA involvement. A formal close-out of audits is done in about half of the companies, either by issuing a close-out memo or a status change in the audit database.

Key performance indicators (KPIs) and key quality indicators (KQIs) for CQA

KPIs were not measured for benchmarking back in 2004. Today, quantitative KPIs are established in most CQA units (83%). Typical examples mentioned in the survey include:

• Number of audits performed

• Timelines of audit reporting

• Number of critical findings identified

While these indicators are easy to measure, their significance for the value CQA adds to the organization is limited. However, KQIs to measure the quality of CQA performance are not common (25% of the participating companies).

Responses for KQIs include:

• Quality assessment of specific CQA activities, like audit performance and reporting

• Feedback of auditees

The data collected on KPIs and KQIs clearly demonstrate that these indicators are not yet well established within the companies. Further development and improvement in the area of meaningful KQIs, including oversight on processes and outsourced activities, is needed-as it is explicitly requested for pharmacovigilance audits (Guideline on Good Pharmacovigilance Practices (GVP) – Module IV; “Evaluation of the quality of audit activities”).¹¹

Conclusions, recommendations

Overall, the current benchmarking showed a positive development of CQA as a key player for quality over the past 10 years. However, we also identified needs for further improvement. Our critical appraisal and recommendations can be summarized as follows:

• CQA is reporting to a higher management level compared to 2004, demonstrating the increased importance of CQA within the organization.

• The data received indicate different organization of QM activities (i.e., they can be centralized in CQA or decentralized in the respective business functions). There are pros and cons for both models. It is recommended that organizations put measures in place to mitigate any disadvantage of the chosen model.

• CQA today covers a broader variety of tasks than 10 years ago (e.g., inspection management, non-compliance investigations, and vendor oversight, as well as involvement in process improvements). Therefore, it is necessary to provide sufficient and qualified resources with in-depth knowledge of both business processes and quality management.

• The growing dependence on outsourcing across business areas increases the need for skilled, internal quality management staff to ensure sponsor oversight.

• The increase of full package outsourcing, including CQA activities, to the same CRO carries the risk of reduction of sponsor control and a potential conflict of interest. Thus, it is recommended to outsource CQA activities to a different service provider.

• The move from site to vendor and system audits is seen positively and reflects the paradigm shift in QM to focus more on processes and systems. However, we highly recommend that site audits not just be de-emphasized but rather that companies consider the right balance of different audit types.

• Higher capacity spent for onsite audits is partly due to increased complexity of trials and related documentation. However, we doubt that further increasing the number of days spent for audits would substantially change the outcome of an audit. The value of an audit very much depends on qualification of the auditor as well as on an optimized audit process.

• CAPA processes have been implemented in the past years. These processes need to be further evolved into robust quality management systems with active and permanent senior management involvement to achieve sustainable results.

• Meaningful KPIs describing CQA activities need to be accompanied by KQIs focusing on the quality achievements delivered by CQA.

Over the past 10 years, we observed a considerable progress in the area of clinical quality assurance. With TransCelerate, pharmaceutical companies started discussions on efficient quality management system across organizations. This is one important positive step in regard to harmonization. Nevertheless, further development towards fully mature QM systems is needed, as also stated in the EMA Reflection Paper on risk based quality management in clinical trials,¹²ICH Q10,¹³ and the ISO 9001-2015 revision.¹⁴

Arthur Hecht, is Global Auditing Manager in Quality Management, Medicine and Regulatory, Boehringer Ingelheim Pharma GmbH&Co. KG; Eva-Beate Ansmann, MD, is an independent consultant for QA/QM consultancy and GCP auditing; Heiner Gertzen, PhD, R&D Clinical & Medical Quality Operations, Sanofi; Evelyn Jäger, PhD, is a Compliance Manager in Quality Management, Medicine and Regulatory; Boehringer Ingelheim Pharma GmbH&Co. KG; Dorette Schrag-Floss, DVM, PhD, is an independent senior consultant for QA/QM and for GCP and GVP auditing; Friedbert Theis, PhD, is Senior Manager Quality & Compliance, Sanofi-Aventis Germany

References

1.    http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2013/11/WC500155491.pdf

2. http://www.transceleratebiopharmainc.com

3.    http://www.ema.europa.eu/docs/en_GB/document_library/Report/2014/06/WC500168824.pdf

4.    http://www.ema.europa.eu/docs/en_GB/document_library/Referrals_document/GVK_Biosciences_31/WC500173714.pdf

5. http://www.appliedclinicaltrialsonline.com/clinical-quality-assurance-benchmarking

6.    http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E6/E6_R1_Guideline.pdf

7.    https://na.theiia.org/standards-guidance/Public%20Documents/IPPF%20Standards%20Markup%20Changes%202013-01%20vs%202011-01.pdf

8.    http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2012/12/WC500136233.pdf

9.    http://www.ema.europa.eu/docs/en_GB/document_library/Report/2011/08/WC500109777.pdf

10. http://www.iso.org/iso/04_concept_and_use_of_the_process_approach_for_management_systems.pdf

11. http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2012/12/WC500136233.pdf

12. http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2013/11/WC500155491.pdf

13. http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q10/Step4/Q10_Guideline.pdf

14. http://www.iso.org/iso/home/standards/management-standards/iso_9000.htm