Monitoring the Monitors

A data and safety monitoring plan (DSMP) is an essential part of any clinical investigation. As outlined in National Institutes of Health (NIH) guidelines^1,2 the Code of Federal Regulations,^3,4 and Good Clinical Practice (GCP) Guidelines,⁵ DSMPs are designed to ensure subject safety and data validity.

Despite their importance, DSMPs, in our experience, are often less well defined than other components of a research protocol, particularly when they are designed by investigators rather than by industry sponsors. Consequently, data and safety monitoring is frequently neglected during the study's duration.

To assist investigators in developing and implementing proper DSMPs, the Partners Human Research Quality Improvement Program (QI Program) developed a set of guidelines, including a checklist, to ensure not only that all necessary elements of a DSMP are included in the protocol, but that they are implemented accordingly throughout the duration of the study.

The purpose of this article is to describe and share these guidelines. They are likely to be relevant to anyone involved in designing, implementing or reviewing clinical research protocols, including investigators, study coordinators, research administrators, and IRB members at academic, medical, and research institutions.

Background

While there is an abundance of published literature concerning Data and Safety Monitoring Boards, the basic elements of DSMPs have gained considerably less attention. A few papers published to date emphasize the importance of developing DSMPs prior to the initiation of a study.^6-8 The complexity of the plan will vary depending on the degree of risk involved; indeed, risk assessment is the pivotal first step in designing the DSMP.⁷ Once established, investigators should be prepared to modify monitoring plans based on findings by the Data and Safety Monitoring Boards, safety monitors, and/or the investigator's monitoring of study subjects.⁸

As far back as 1979, NIH issued a policy stating, "every clinical trial should have provision for data and safety monitoring."⁹ In 1998, NIH published guidelines describing necessary elements for adequate data and safety monitoring for clinical trials.¹ This document indicated that depending on the nature, size, and complexity of a study, various types of monitoring could be anticipated to evaluate the data quality and safety of the subjects.

Up Close: What a Good Data and Safety Monitoring Plan Looks Like

It further recognized that the principal investigator may be responsible for monitoring investigator initiated Phase I studies, while a committee of experts (e.g., a Data Safety Monitoring Board) may be responsible for Phase III studies. In 2000, NIH issued further guidance for Phase I and Phase II studies, including who is responsible for safety monitoring.² In addition, the guidelines also stated that at a minimum, DSMPs should include a description of how adverse events will be reported to the Institutional Review Board, FDA, and NIH.

The FDA regulations require that each clinical investigator is responsible for ensuring that a study is conducted according to the signed investigator statement, the investigational plan, and applicable regulations.^3,4 The FDA also recommends that written procedures for monitoring clinical investigations be established to assure the quality of the study and that each member involved in the monitoring process carries out his/her duties.¹⁰ Furthermore, it recommends standardized general procedures that are supplemented by more specific monitoring procedures tailored to the individual study.¹⁰

Despite the requirements outlined in the Code of Federal Regulations, FDA warning letters commonly cite investigators for failing to follow the signed investigator statement and the investigational plan. For example, investigators have been cited for not conducting follow-up safety procedures within specified windows, failing to report adverse events to the appropriate regulating bodies, and enrolling subjects who did not meet inclusion/exclusion criteria.¹¹ These warning letters suggest that while the investigator may not be able to alter the provided industry developed protocol, a less well-defined DSMP does not absolve the investigator from responsibility for ensuring subject safety and data integrity.

The QI Program Handy DSMP Checklist

The general responsibilities of sponsors set forth in the federal regulations require proper monitoring of the investigation with assurance that it is conducted in accordance with the investigational plan and protocol.^3,4 Therefore, investigators who hold an Investigational New Drug Application or Investigational Device Exemption and act as sponsors have the additional responsibility to establish monitoring procedures.

GCP guidelines⁵ recognize that the study sponsor is responsible for adequately monitoring trials. Sponsors also should determine the appropriate extent and nature of the monitoring based on factors such as study size, complexity, and design. The protocol should outline plans to evaluate safety by incorporating the following measures into the protocol: safety parameters; methods and timing for assessing, recording, and analyzing safety parameters; procedures for recording and reporting adverse events and intercurrent illness; and the type and duration of the follow-up of subjects after adverse events. GCP guidelines also recommend a section on data handling and record keeping in the protocol, but do not specify what this section should include.

Protocol Detective: DSMP Checklist

Elements of a DSMP

An adequate DSMP should address four areas: safety monitoring, data monitoring, product accountability, and privacy and confidentiality. Each area includes a related set of elements. Table 1 illustrates these elements with examples.

Safety monitoring. Safety monitoring refers to the vigilant observations required to minimize threats to the safety and welfare of research subjects. A safety monitoring plan will be unique to each study; the monitoring activities are dictated by study-specific risks. In determining the type and frequency of monitoring required, sites should consider factors such as the study's purpose, design, population, procedures (including investigational products), sample size, and trial phase.

Three essential elements make up a proper safety monitoring plan: important safety information, stopping rules, and adverse event reporting. Safety monitoring itself should include collecting and subsequent monitoring of specific information/data that would indicate a threat to a subject's health and welfare. Parameters for each safety observation should be clearly specified in measurable terms. The frequency of safety observations should be defined for each variable and should be such that researchers are able to identify clinical or subclinical problems at an early stage. The person(s) responsible for reviewing safety data should be clearly identified. The FDA expects that the monitor "need not be a person qualified to diagnose and treat the disease or other condition for which the test article is under investigation, but somewhere in the direct line of review of the study data there should be a person so qualified."¹⁰

Stopping rules include conditions under which an individual subject's study participation will be terminated, as well as conditions under which the entire study should be stopped or modified. The principal investigator is typically responsible for decisions of subject termination, whereas a Data Safety Monitoring Board or safety monitor may be responsible for making recommendations to stop and/or modify the study, depending on the phase of the investigation.

The last essential element of the safety monitoring plan is an adverse event reporting plan. All studies are subject to unique institutional adverse event reporting requirements. At a minimum, the DSMP should indicate who will be responsible for reporting adverse events at the institutional level. In addition, adverse events occurring in clinical trials involving investigational drugs or devices must be reported to the FDA by the sponsor in accordance with relevant federal regulations. In the case of multisite studies, the coordinating center is responsible for reporting and distributing safety information to participating sites.

Data monitoring. In contrast to safety monitoring, the purpose of the data monitoring plan is to ensure both adherence to the approved clinical investigative plan and the validity of the data collected. Its specific elements can include review of informed consent documentation, confirmation of subject eligibility, and verification of the accuracy of data transfer from source documents to case report forms.

Data monitoring should be scheduled at intervals sufficient to keep up with subject enrollment and/or the expected volume of subject information. Unlike the safety monitoring plan, qualified study staff (not necessarily the principal investigator), such as a research nurse or study coordinator, can be assigned responsibility for conducting and/or overseeing data monitoring. In many cases, the person responsible for data review reports on a weekly or monthly basis to a principal investigator.

The plan should additionally clarify how protocol deviations will be handled, such as filing appropriate reports to the IRB and instituting corrective action to minimize recurrences. It should be emphasized that despite qualified staff assignments to oversee data monitoring, the principal investigator remains ultimately responsible for data quality and integrity.

Product accountability. In studies that involve an investigational product, a plan for product accountability should be in place to ensure that all study drugs, biologics or devices are accounted for. This plan should provide for a record of receipt, distribution, and disposal of the product, as well as an overall inventory accounting for all investigational products.

A person should be assigned responsibility for product accountability. Even if an institution-based research pharmacy assumes responsibility for these functions, the investigators are still ultimately responsible for product accountability and should still maintain records that document subject receipt of specified doses of the investigational product and return of any unused product. Devices can pose a greater challenge to research sites because the clinical research sites are typically required to manage accountability themselves.

Privacy and confidentiality. In addition to plans to protect subject safety and adherence to the protocol, sites should consider plans to protect subject confidentiality. Breaches in confidentiality have the potential to lead to unanticipated problems, ranging from embarrassment to socioeconomic impacts (e.g., loss of insurance or employment).

The HIPAA Privacy Rule, which took effect in April of 2003, requires that individually identified information (protected health information) created or collected during a study must be coded or de-identified.¹² Legal consequences associated with violation of the Rule include civil fines and criminal penalties.¹³

At a minimum, the DSMP should address how data will be protected, including how they are stored (password protected electronic records and/or paper files), who has access to the files, and what steps will be taken to prevent unauthorized access (e.g., contained in a locked office or large multistudy storage room). In addition, the plan may need to consider how investigators will protect subject privacy and confidentiality if the protocol calls for data or tissue samples to be shared with collaborators for the current or future research.

Implementation

To ensure compliance with FDA regulations, NIH policies, and GCP guidance, the QI Program developed a checklist that includes essential DSMP elements (Figure 1). Its main advantage for QI purposes lies in helping identify missing or underdeveloped DSMP elements when reviewing study protocols for study site initiated or IRB requested onsite review.¹⁴ Deficiencies can be easily detected, resulting in effective recommendations for corrective action.

The checklist also serves an important educational purpose. Protocols for studies funded by government agencies, private foundations, and intra-institutional sources often lack the detailed DSMP descriptions characteristic of those developed by industry sponsors. To help investigators prepare more comprehensive DSMPs or improve existing plans in ongoing studies, we have conducted educational sessions to introduce the checklist. Investigators can use it as a template, supplemented by additional monitoring procedures specific to their study. Also, IRBs may use the checklist as a basis for evaluating the adequacy of DSMPs in protocols submitted for both initial review and continuing review. To standardize expectations for investigators or improve consistency in protocol reviews, some IRBs may wish to incorporate such a tool into institution-specific guidance documents available to their members as well as to the research communities they serve.

Despite long-standing policies and available regulatory guidance, evidence found in FDA warning letters and reflected in our own experience indicates a persistent gap between DSMP regulatory expectations and reality. The FDA citations discussed earlier, illustrating that investigators commonly fail to follow the investigative plan, are just one example of this gap. Adequate monitoring would likely have identified lapses in protocol adherence, prompting corrective responses. It may also be necessary and appropriate for sites to exercise additional monitoring beyond that provided and/or required by the industry developed protocol. Investigators should use their judgment to decide if the protocol's DSMP is not as well defined as it should be.

User-friendly tools are needed to help investigators, sponsors, IRBs, and quality assurance/improvement staff alike to translate regulating requirements into practice. The DSMP checklist presented here offers concrete guidance to help users incorporate all expected elements of data and safety monitoring into research protocols. It further provides a basis for evaluating regulatory compliance and improving study site performance.

To be effective, DSMPs must be developed and in place prior to the onset of the clinical trial and modified as needed as the study progresses. Given the importance of DSMPs, tools such as the checklist described here can help to ensure regulatory compliance while promoting subject safety and data accuracy/completeness. We believe that this tool responds to a significant need for improved accountability as clinical trials come under closer scrutiny in both public and professional arenas.

References

1. National Institutes of Health, NIH Policy for Data and Safety Monitoring (NIH, Bethesda, MD, 1998).

2. National Institutes of Health, Further Guidance on a Data and Safety Monitoring for Phase I and Phase II Trials, (NIH, Bethesda, MD, 2000).

3. Code of Federal Regulations, Title 21, Part 312, Sections 50, 60 (US Government Printing Office, Washington, DC).

4. Code of Federal Regulations, Title 21, Part 812, Sections 40, 100 (US Government Printing Office, Washington, DC).

5. International Conference on Harmonization, "Good Clinical Practice: Consolidated Guidance (ICH-E6)," April 1996.

6. N.T. Artinian, E.S. Froelicher, J.S. Vander Wal, "Data and Safety Monitoring During Randomized Controlled Trials of Nursing Interventions," Nursing Research, 53 (3) 414–418 (2004).

7. L. Slimmer and B. Andersen, "Designing a Data and Safety Monitoring Plan," Western Journal of Nursing Research, 26 (7) 797–803 (2004).

8. M.L. Terrin, "Evaluating and Implementing Data and Safety Monitoring Plans," Journal of Investigative Medicine, 52 (7) 459–463 (2004).

9. NIH Guide for Grants and Contracts, Volume 8, No. 8 (1979).

10. Food and Drug Administration, Guideline for the Monitoring of Clinical Investigations (FDA, Rockville, MD, 1998).

11. U.S. Food and Drug Administration, FDA's Electronic Freedom of Information Reading Room—Warning Letters and Responses, http://www.fda.gov/foi/warning.htm (Accessed January 2007).

12. Code of Federal Regulations, Title 45 CFR Section 164.514 (U.S. Government Printing Office, Washington, DC).

13. Health and Human Services, OCR Privacy Brief, Summary of the HIPAA Privacy Rule, http://www.hhs.gov/ocr/privacysummary.rtf (Accessed May 2007).

14. D. Wolf and P. O'Rourke, "Ensuring Investigator Compliance and Improving Study Site Performance: Implementing a Quality Assurance and Quality Improvement Program in Academic Health Centers," Clinical Researcher, 2 (5) 2–14 (2002).

Sarah White* is a senior quality improvement/assurance specialist, QI Program, Partners Healthcare Systems, 116 Huntington Ave, 10th Floor, Boston, MA 02116, email: swhite12@partners.org. Lucy Feild, PhD, RN, is a senior quality improvement/assurance specialist and Delia Wolf, MD, JD, is director of the QI Program at Partners Healthcare Systems.

*To whom all correspondence should be addressed.