Value-Based Auditing of Clinical Trial Software Suppliers

Why do we audit our suppliers and what do we hope to achieve when we do? Certainly, regulated companies need to ensure their systems meet both business and regulatory requirements, which include systems provided by third-party suppliers.

To meet the rapidly evolving needs of regulated companies, many technology suppliers have adopted advanced development, implementation and hosting methods. All too often, however, unprecedented and unfamiliar methodologies leave these same regulated companies unsure of how to audit in a way that is sufficient for purpose and compliant with regulatory expectations and their own procedures. Thus, audit practices need a facelift to keep pace with the technologies that need to be assessed.

While some regulators have a reputation of not accepting new technologies, in mid-2012, the U.S. Government Accounting Office identified a number of practices and approaches as effective for applying Agile software development methods to IT projects. Government officials who have used Agile methods positively commented on the effectiveness of these practices.¹ More recently, in mid-2013, the U.S. Federal Risk and Authorization Management Program (FedRAMP) announced its approval of a cloud technology provider for use in government business, under an assessment sponsored by the U.S. Department of Health and Human Services (HHS).² Although these notices are specific to the U.S. and do not explicitly reference the FDA, it is not a great leap to envision other government entities within the U.S., as well as other healthcare regulators worldwide, recognizing the value and necessity of considering new technologies as they look to improve their own operations.

Audit Challenges

Supplier audits are generally focused on compliance with regulatory requirements by assessing the adequacy of the overall quality system, validation, training, security and privacy, and product and service quality. The following section examines a sampling of challenges associated with these practices and the overall auditing process:

• Maintaining Focus on Quality. Any reputable supplier will acknowledge that it is quite possible that valid and perhaps even substantive observations or findings are uncovered and reported by auditors. However, some observations are so obtuse and so unimportant (in the bigger scheme of things) that it begs the question: What are auditors assessing? Despite that every auditor aims to assess quality, sometimes their focus shifts to minutia, misrepresenting facts, or failing to respect supplier policies or practices.

• Working with Third-Party Auditors. An outcome of corporate downsizing and the trend towards outsourcing activities is that regulated companies increasingly rely on the ever-growing community of third-party auditors. Companies should perform due diligence on these third-party auditors to ensure that the auditors are knowledgeable about the new technical environments they are being tasked with assessing.
  
  Not only do regulated companies outsource more and more of their non-core operations; technology suppliers do it too. For a Software-as-a-Service (SaaS) supplier, this might mean outsourcing hosting operations to Infrastructure-as-a-Service (IaaS) and Platform-as-a-Service (PaaS) providers, as well as outsourcing other generic systems and services—for example, source code management system, electronic document management system (eDMS), learning management system (LMS)—to other SaaS providers. Although there isn’t always a formal transfer of study obligations from a regulated company to a SaaS provider, the expectation is that each party will perform an appropriate due diligence assessment of their suppliers.

• Regulatory Inspections. Suppliers recognize that it is in the best interest of all parties (e.g., sites, sponsors, CROs, suppliers) that actual regulatory inspections of these various parties go well and will do what is necessary to support such inspections. With this in mind, regulated companies (and their auditors) should also appreciate that requests (demands) made by regulatory authorities carry significant weight with suppliers, and it should be understood that a request refused by a supplier during an audit by the company does not necessarily mean a refusal during a regulatory inspection.

• Traceability. Regulators expect traceability from requirements through to testing. Historically, with Waterfall software development, this requirement was addressed via a standalone Traceability Matrix. It is important to note, however, that the regulatory expectation is for traceability and not necessarily a Traceability Matrix—as long as one can demonstrate traceability via another mechanism. The use of automated testing tools is accompanied by test plans and test output that look very different from that associated with traditional testing approaches. Suppliers should be prepared in advance to explain how newer, more scalable development and testing approaches can still be used in a regulatory compliant manner with regulatory compliant deliverables and how, in fact, the use of such approaches can result in higher quality solutions.

• Electronic Documentation. Suppliers who are subject to many client and prospect audits have options for how they present information to auditors. Historically, the expectation from auditors was that the supplier would wheel in multiple carts of documentation, including copies of quality system (QS) documents (e.g., policies, SOPs, work instructions); validation deliverables; test plans, scripts and results; and training records. The more progressive suppliers have adopted electronic systems for creating and maintaining the aforementioned documents because such systems introduce effectiveness and efficiency into their daily operations.  Auditors who demand (or attempt to demand) that the supplier revert to supplying paper copies of such documents are actually doing themselves a disservice, because they are not assessing how the supplier is actually managing their operations. The advantages associated with the use of electronic documentation are addressed further under Remote Audits below.

• Audit Location. Many international companies, even those that have grown through acquisition, operate globally and have global standards and procedures. For this reason—along with the fact that key subject matter experts might be centralized in a particular location or that the staff responsible for hosting audits are located in a particular office—such suppliers often host audits in only one location. The idea that a regulated company needs to qualify each individual supplier office is often based on provisions in that company’s Supplier Audit SOP and not because there is any real value to complying with such provisions. Suppliers may determine that it is in their own best interest, as well as in the interest of their clients and prospects, to host audits from multiple locations. However, companies are encouraged to respect supplier policies and procedures for audit hosting, as such policies and procedures are likely designed to ensure successful audits for all parties.

• Audits by Customers of CROs. Many suppliers have taken a firm position that they will only accommodate audits by direct clients or prospects. Such a position is not taken lightly. Many of these suppliers routinely host more than 100 on-site client/prospect audits on an annual basis. Extending audit rights to customers of CROs has the potential to increase the number of audits hosted many times over.  Furthermore, the supplier is unlikely to have a contractual arrangement with the CRO’s customer and may not be responsible for associated services that the CRO is providing. Thus, the supplier would not be able to answer all of the questions raised by the CRO’s customer.

• New Technology. Progressive SaaS providers have replaced the traditional Waterfall software development approach with an Agile development approach, along with greater dependence on automated testing tools. Nonetheless, the use of Agile and automated testing often presents challenges for auditors who are not used to (and are uncomfortable with) anything but Waterfall development and the associated deliverables.

Additionally, SaaS providers offering single instance multi-tenant (SIMT) solutions means that all users are upgraded to a new version of the software at the same time. As with Agile, this change also presents challenges for auditors who are familiar with reviewing validation documentation for a specific version of the software. Instead, the SaaS provider should be prepared to demonstrate (and auditors prepared to assess) a strong change control process as a means of ensuring that the version of the software being deployed is well managed and that any and all versions of software used at any particular time are clearly identifiable.

Audit Alternatives

Many regulated companies and suppliers are now supplementing traditional on-site audits with alternative auditing approaches, thereby decreasing the frequency of on-site audits and increasing knowledge transfer between organizations to improve audit quality. Several approaches are being adopted, including conducting supplemental audits with (or instead of) periodic on-site audits remotely, thereby reducing the high cost of face-to-face interactions. The following section examines a sampling of audit alternatives:

• Remote Audits. Rather than always conducting on-site audits of key suppliers, regulated companies are accepting (and in some cases requesting) that the audit be conducted remotely. A desire to reduce overall audit travel costs certainly contributes to such requests. In addition, remote audits enable involvement by individual audit participants whose schedules might not accommodate participation in multiple on-site supplier audits. In order for such audits to be successful, the supplier must be able to present relevant records electronically. Suppliers are now also exploring how they can facilitate audits that might span several weeks versus several days by giving audit teams remote access to selected artifacts over a much more extended period of time. Suppliers who have adopted electronic solutions for document management (including quality system document management), automated testing and electronic portals for maintaining validation artifacts are thus in a much better position to facilitate remote audits.

• Group/Joint Audits. Some suppliers are considering requests to accommodate audits by representatives from a consortium of regulated companies who conduct an audit together and prepare the audit report as a group. Such audits have the potential to result in reduced time and effort for all parties, especially if participation in the group audit means limitations on individual company audit expectations and rights. Additionally, the collaborative environment has the potential to improve the overall quality of the audit experience because companies have a forum to pose questions collectively, whereby every company benefits from hearing the response. This model has not been more routinely adopted because there are real challenges in establishing the necessary contractual arrangements between the regulated companies, a third party and the supplier. These parties need to determine who will manage the audit and audit follow-up, including the distribution of the audit report response. This auditing option might be more advantageous for certain segments of a supplier’s client base where there are geographic challenges for each individual company conducting their own on-site audit.

• Audit Webinars. Supplier-hosted audit webinars broadcast the audit presentation and question and answer session to a group of participants who are located anywhere around the globe. Audit webinars allow auditors from multiple regulated companies to collectively participate remotely in an audit that focuses on a particular supplier solution or module, or on particular aspects of the supplier’s operation (e.g., the supplier’s QS, the supplier’s Agile development methodology). These webinars contribute to a reduction in overall travel costs, provide flexibility and convenience and increase knowledge sharing. Furthermore, suppliers are encouraged to record such audit webinars and make these recordings available to their clients and prospects. In addition, the supplier might consider opening up audit webinars to non-direct clients (such as customers of CROs) who otherwise would not have the opportunity to audit the supplier directly.

• Third-Party Certifications. Third-party certifications, such as Service Organizations Control (SOC) 2 or U.S. Federal Information Security Management Act (FISMA), can serve a useful purpose in demonstrating the robustness of data centers. Many suppliers offer copies of the reports associated with such certification for review by regulated companies.

• Videos and Virtual Tours. Videos and virtual tours can be leveraged as a tool in advance or instead of an on-site audit. These user-friendly modes of information sharing are especially useful in audits, and can also be used to provide others within the regulated company an easy-to-understand “picture” of how the supplier operates.

• Testing Approaches. Suppliers would be well served by being more transparent about their testing approaches and test results. Regulated companies and the regulators themselves generally agree that there is minimal value gained by retesting what the supplier has already tested. However, if regulated companies are not knowledgeable about the depth and breadth of testing performed by the supplier, they may choose to duplicate much of that testing in their own user acceptance testing (UAT).
  

Conclusion

To explore newer, more productive ways of complying with regulatory expectations regarding supplier assessments, regulated companies and suppliers need to work together. This will allow each organization to do what it does best to achieve its business and regulatory goals. These same suppliers and regulated companies can also provide examples of more constructive supplier audits that might well identify problems or potential gaps, but simultaneously present joint opportunities for improvement.

Frances E. Nolan, Vice President, Quality & Regulatory Affairs, Medidata Solutions

References

1. http://www.gao.gov/products/GAO-12-681?goback=%2Egde_132741_member_139589417

2. http://www.washingtonpost.com/business/technology/amazon-web-services-gets-government-approval-for-federal-cloud/2013/05/21/df8c563a-c20c-11e2-8c3b-0b5e9247e8ca_story.html