Developing and Validating Electronic Diaries

Subject diaries are used to collect primary and secondary endpoints in about 25% of all clinical trials.1 In recent years, the traditional paper diary booklets have been replaced in some instances by small electronic devices for recording diary information.2 Our series of articles is aimed at helping clinical researchers understand the development, implementation, and regulatory compliance of these electronic patient diary systems. Our particular focus is on the reasonable apprehensions expressed by clinical teams considering whether to use electronic diaries. They worry about these issues.

• Lack of clarity about complying with the regulations governing electronic systems (discussed in Part 2)

• Lack of knowledge about how an electronic system is specified, built, and deployed

• The effect of including an electronic diary in the clinical trial development process and timelines

• The number of work process changes that could be required of participants in the clinical trial process: sponsor staff, monitors, investigators, and subjects.

To address these concerns, this series presents the clinical, technical, and regulatory issues involved in the development and implementation of electronic patient diary (EPD) systems that meet regulatory requirements. Part 1 compared paper and electronic methods in the context of existing FDA and ICH regulations and guidance for capturing diary data.3 We concluded that newer, electronic methods are in fact more compliant with FDA and ICH data standards than traditional paper diaries. In Part 2, we began discussing the clinical and technical considerations for developing an EPD system. We focused on the clinical protocol and its definition of the medical moments-the health experiences-to be recorded by the subject, and we concluded that it is key to the success of an EPD system.4

A Computerized System

In Part 3 we discuss the development and validation of an EPD system based upon a clinical protocol. As shown in Figure 1, a computerized EPD system extends beyond the hardware and software used by the subject. It also includes the systems components and work processes for all users of the system; the core of the EPD system is the diary device containing protocol-specific software. The device also requires some type of communication software and hardware-for example, a peripheral wired modem that connects to a regular phone line-to transmit the subject data to a central data repository or server.

Figure 1. Computerized system for electronic diaries.

Once it is on the server, the subject's data can be processed and viewed in a secure environment by those responsible for trial management and conduct. The investigative site can use this information to evaluate the subject's compliance with protocol procedures. The sponsor's clinical team can review the data from the server to monitor the investigative sites' activities and to assess the subjects' performance. The components of a successful EPD system include the activities carried out by the subjects and the trial managers as well as the hardware and software.

The key players in developing an EPD system are the clinical team responsible for the trial and a technical team that brings the EPD technology and the expertise to customize the technology for a particular trial. To design and develop an EPD system that meets the needs of the protocol takes close collaboration between those two teams. Many aspects of the collaboration should be familiar to the clinical team, given that every trial requires a well-coordinated effort between the sponsor, CRO (where appropriate), investigative sites, and subjects.

Glossary

Unique features of the collaboration of the clinical and technical teams are the development of the EPD technology and the support structures-training materials, for example. This development, however, should be greatly facilitated by the technical team's expertise. The result should be that the experience of developing an electronic diary system and then executing the trial in which it is used should feel much like that of any trial for a clinical team.

A successful collaboration of the clinical and technical teams depends upon clearly defined responsibilities, tasks, and deliverables. Figures 2 and 3 diagram the tasks required to develop, implement, and eventually retire an EPD system in the context of a clinical trial. Figure 2 illustrates the system development life cycle (SDLC), activities primarily driven and executed by technical team. Figure 3 depicts an EPD clinical trial life cycle (CTLC), which summarizes the activities and support structures needed, mapped onto the typical steps involved in planning, launching, executing, and closing out a trial. CTLC activies are driven by the clinical team and executed through collaboration of the clinical and technical teams.

In Part 2 of this series we discussed Steps 1 and 2 of these life cycles, the development of a clinical protocol and the user requirements specification (URS) for an EPD system. Assuming a clear set of specification documents for the system, we focus here on Steps 3 through 6 of the SDLC and CTLC, or the preparation of EPD system for trial launch. The two streams of activities that occur during preparation are the building and testing of the EPD software (SDLC), and the development of all of the support infrastructures for the software (CTLC).

Details of the tasks, responsibilities, and deliverables at each step are described below. As in any GCP study, an essential component of all of these activities is careful and clear documentation of the process. Figure 4 diagrams the computerized system validation (CSV) documentation that results from this documentation. The outcome of Step 6 in the system development life cycle and the clinical trial life cycle (diary acceptance) is a validated electronic diary system that performs as intended.

CTLC Step 3

Infrastructure (Paty, clinical-regulatory) Once a protocol is approved, the clinical team begins to establish the infrastructure for the study. This includes selecting a CRO, if one is to be used, and sites for the study. The sites do not need to be technically proficient if the EPD system is designed with end users in mind. In our experience the most important site qualifier is willingness to learn something new. Of course, if the site has previous experience with an electronic diary system, this can significantly aid the implementation of the trial. For many EPD systems, the infrastructure requirements for sites are minimal: a regular phone line and Internet access.

Figure 2. The System Development Life Cycle (SDLC).

Equipment selection and purchase is another component of preparing for an electronic diary system. With the technical team's advice, the clinical team makes the final decision about the hardware best suited for the subject population being recruited for the trial. Subject hardware includes the data collection device and possibly communications hardware (an attachable wireless or wired modem, for example).

Investigative sites may need specific hardware to support the system, but for systems that rely on the Internet, sites may simply require Internet access and standard browser software. The same applies to the sponsor and/or CRO staff. For global trials, the subject hardware selected needs to support multiple languages. Communications device needs may differ from country to country given variability in wired and wireless capabilities in Europe, Asia, and South America. Once the hardware decisions are final, early procurement ensures that equipment is not a bottleneck in developing and launching the trial.

Diary design (Stokes, technical-regulatory). Steps 1 and 2 in the EPD system development life cycle translated the clinical protocol into a user requirements specification (URS). The technical team's first action in diary design is to translate the user's view of a diary (their URS) into the functional requirements specification (FRS), a technical document that describes the EPD system in a way that a programmer knows what has to be built (Figure 2, Step 3). To maintain the integrity of the original protocol throughout the validation of the electronic diary, the technical team creates a traceability matrix that links the requirements in the protocol to those in the URS and then to the FRS. The traceability matrix has a central role in ensuring that anyone can begin with the protocol and trace through all of the documents to ensure the system was built as specified.

Traceability matrix. It is important to create a traceability matrix showing how the URS addresses the Protocol sections and how each and every URS statement is addressed by one or more FRS statements. Here are some examples of protocol statements with corresponding URS and FRS statements.

Protocol statement (from clinical team). "Weekly Assessment: The following question will be assessed approximately every seven days during treatment period as part of the subject's daily assessment . . ."

URS statement 24 (from clinical team). Flow chart made to show Q&A dialogue for daily assessment and also the insert prompt for a weekly summary assessment.

FRS statement 53 (URS 24) (from technical team). Screen prompt question is "Answer the following item according to how you have felt during the past week." This prompt and subsequent dialogue is to show in the EPD on study days 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, and 77.

The system design description (SDD). In order to tell the programmer how to build the diary, the technical team converts the FRS statements into a system design description (SDD). Usually a design description statement covers more than one FRS statement. Thus, an item-for-item traceability to the SDD is not useful, but an overall system design review should be performed to ensure that all the FRS statements have been covered somewhere in the system design. Taking the FRS example described above, the appropriate SDD statement (from technical team) would be

Figure 3. Electronic patient diary clinical trial life cycle (CTLC).

Code the system with a study-day tracking variable (variable SD4) so that the EPD tracks the current Study Day number. Code the Daily Assessment to always check variable SD4 whenever it is called in the code and to check if the Study Day (SD4) is day 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, or 77. If Study Day matches, then code system to display extra dialogue item as per UR24. If no match is made, then code so that no extra display is issued to the EPD user.

The process of moving from protocol through to functional requirements and software design is usually an interative one. It can be readily accomplished with consistent and effective communication between the clinical and technical teams. After several EPD trials, when the clinical team learns the capabilities of the technology, it can refine the protocol statements to more easily translate into functional requirements. The technical team can then develop standard diary modules and the whole EPD development process becomes faster and more efficient.

CTLC Step 4

Software review, training, and plans (Paty, clinical-regulatory) With the sites selected and the technical system specifications written, the clinical team-with considerable support from the technical team-can focus on planning and implementing various components of the EPD system. They can

• Review and approve EPD software prototypes

• Develop EPD data management plans

• Develop and approve training materials

• Develop a help desk plan and approach

• Develop a plan for deploying units during the study.

Software review. A very important component at this stage of development of the EPD system is a clinical team review of alpha (early stage) software. This review allows the clinical team to determine whether the technical team is developing software that accurately reflects the protocol. Most important, it is an opportunity for the clinical team to determine whether the software is "coming alive" in the way they had expected. If it is not, this is the time to pause and reconsider the design of the system and its specifications. When the clinical and technical teams collaborated during the specification process, the typical outcome of the alpha review is making only a few minor changes to the system.

Training materials. In an EPD study, concise and clear training is essential for all users of the system. 21 CFR 11.105 states that persons using an electronic data capture system need to be appropriately trained on the system. Training materials need to be developed for the different user groups-subjects, the investigative site staff, sponsor/CRO clinical research associates (CRAs), and other study management staff. The technical team often provides a core set of training materials that are customized for the specific trial. The customizing requires knowledge of the protocol and the system, suggesting a clinical-technical team collaborative effort.

Help desk. Another component important to the planning of an EPD trial is a plan for a technical help desk that will support the study. The clinical team's input is important here, because the technical help desk will be dealing with the subjects, the investigative site personnel, and the CRAs. As such, help desk personnel will need to understand how to address technical issues that may be specific to the EPD protocol. The technical team provides the technical details, and the clinical team provides protocol-specific content. That content is then reflected in scripts for the trial that guide help desk personnel who respond to questions from the trial participants. In multinational trials, the scripts can be developed so that they can be readily translated to all relevant languages. Implementing the scripts and general management of the help desk is typically the responsibility of the technical team.

Figure 4. Electronic patient diary computerized system validation (CSV) package model.

Equipment deployment. The clinical and technical teams will plan the provisioning (loading software onto the EPD units) and shipping of equipment to investigative sites. Their plan needs to take into account the exact provisioning process, who will perform the activity, how and when units will be shipped to sites, and the timing for returning units from the investigative sites. Given that some trials are conducted with thousands of subjects in several countries, a well conceived plan for provisioning and shipping is an important component of trial preparation. The clinical team will provide input based upon knowledge of the sites' locations, and the technical team will delineate the process and manage the provisioning and shipping process.

Data management. The data management and statistical groups may begin their planning at this point. Based on the database design described in the specifications, the sponsor-CRO data management team can work with the technical team to ensure that the method and format for delivering the data is clear. This will ensure that data can be delivered regularly to the sponsor or CRO for inclusion in the clinical database.

CTLC Step 4 summary. Once the planning, development, review, and approval activities have established the support stucture for an EPD study,

• the materials are prepared to train the system users.

• the plan is in place to support the users if/when they have questions or problems.

• the plan is in place for getting the EPD devices to sites for subjects.

• the plan is in place for delivering data to the sponsor.

The clinical team is ready to initiate sites for the trial, hold the investigator meeting, and launch the study.

SDLC Step 4

Build diary (Stokes, technical-regulatory). Among several components an EPD system (see Figure 1) that the technical team will develop or configure are study-specific versions of software for

• subjects to use in recording their medical moments and transmitting them to the central database server.

• the investigative site to use in reviewing subject data and resolving queries on the central database server.

• the sponsor's clinical team to use in reviewing subject data on the central database server and raising any queries with the investigative sites.

• providing interim and final reports of data to the sponsor and investigative sites.

Coding the EPD software. Two key factors determining the coding and testing time of the EPD device and web software are the structure of the system and the clarity of the documentation. Some electronic diary systems facilitate development by providing predeveloped and pretested pieces of code that can be rearranged (or configured) to suit the trial needs. Only unique requirements for a specific trial would then need to be directly programmed. This configurable approach to building diaries can save both time and money.

The impact of clear documentation cannot be overemphasized. When insufficient time is taken to be clear about Steps 1–3 of the SDLC, programmers get confused directions from the SDD and the process of building the electronic diary is interrupted by false starts and misdirected results that require repair and rebuilding. This causes multiple updates through Steps 1–3 and repeated cycles of testing in Step 5. Such later-in-life problems are costly to repair, and they disrupt the clinical study work process itself. Software engineering literature tells us that changes in the early stages of a project-while the requirementsand architecture are being developed-are much less costly-perhaps 50 to 200 times lower cost-than the same changes made at the implementation and maintenance stages.

Programmer unit testing of the EPD code and device. Programmers test code as they develop it, and this is usually called unit testing. Unit tests also include trying out the EPD code on the selected handheld device. Unit tests are successfully completed prior to release of the system code to formal quality control testing in SDLC Step 5. Code reviews are performed during the diary build step to determine that programmers have followed the coding conventions and other project SOPs. Code reviews also check to determine that a programmer has addressed all relevant requirements according to the FRS and SDD for the section of the code under review.

Writing the system documentation. During the diary build step, documentation is developed for the EPD software, the EPD device, and the central database server components. Reviews are performed to ensure that docu-mentation for system support and user support are written in understandable, standard language.

CTLC Step 5

Study staff preparation (Paty, clinical-regulatory). At this point, the EPD software development is moving into its formal testing period. The software is at a beta stage of development, meaning that all of the necessary features have been built into the system. It is appropriate for the clinical team to undertake another review of both the electronic diary device and Web software to ensure that the overall look and feel of the system matches expectations.

This is also the perfect time to train the trial managers, especially the CRAs, on the system. In many cases, this will be the first EPD trial for a CRA, who has responsibility for monitoring the sites for protocol performance. Pretraining CRAs is important, as it allows them to understand the role they will play, to learn how to monitor an EPD trial, to be knowledgeable when it is time for them to train the investigative staff during the investigator meeting, and to test the study support structures, especially the training materials.

SDLC Step 5

Diary design test (Stokes, technical-regulatory). The quality control (QC) members of the technical team perform formal testing on the diary and its associated system documentation. The focus of this formal quality control testing is to be sure that the EPD was developed in a way that adequately addresses all the functional requirements in the software design. This testing answers the question, "Did we build the diary right to fit the design requirements?" QC testing also confirms that the online help and other system documentation accurately describe the way the system looks and performs.

To execute QC testing, first a test plan is written; it is based on the approved requirements (FRS), software design (SDD), and system documentation. Management approves the test plan, and then persons other than the programmers must prepare test documents. Test scripts are written for topics taken from the SDD and FRS. The traceability matrix is updated to cross-reference the test scripts that evaluated various FRS statements. To have independence of review during the formal testing, it is important that the programmers/builders are neither the test scriptwriters nor the testers. The programmers have already done their own testing with their inside knowledge of the code at the unit test phase in Step 4.

The testing strategy and results are described in a formal test plan and test summary report that are approved by management. The exit criterion from this step is a signed test summary report that describes in full

• how the test plan was executed.

• the testing environment (EPD software, devices, server communications, and systems documentation).

• the size and scope of testing with traceability matrix to FRS.

• any problems that were found, and how they were resolved.

• the comprehensiveness of the testing,

• an overall pass/fail status for the EPD system.

• a management control statement that releases the EPD to the clinical team for user acceptance testing.

CTLC Step 6

Accept diary, test user acceptance (Paty, clinical-regulatory). The EPD system is now ready for final review and approval by the clinical team in the form of user acceptance testing (UAT) of the various system components. UAT evaluates whether the EPD device will work in the context of the subject's daily life, and whether the Web software meets the requirements of the investigative sites and other trial managers. The clinical team also evaluates the training materials and other relevant processes at this stage. There is some formal testing process around this activity for the clinical team, but the focus is on whether the system that was built meets the scientific and clinical objectives of the study as laid out in the protocol. The technical team often provides a standard set of materials to enable clinical teams to test user acceptance quickly and efficiently. If all the previous steps in the process have been carried out collaboratively and documented carefully, the UAT should be a straightforward final review leading to approval of the system before the study launch.

SDLC Step 6

Accept diary (Stokes, technical-regulatory). When the technical team has successfully completed its formal testing of the electronic diary to its FRS and SDD, it is time for the clinical team to test user acceptance of the system. The test strategy is now based on the user requirements specification (URS), the user training materials, and the practical, physical, and logical needs of the trial's work process in the investigator, sponsor, and subject environments.

User acceptance testing answers the question, "Did we build the right diary for subjects to use in this trial?" Acceptance test scripts exercise the EPD using the normal work process for the sponsor, investigator, and subject in the trial. Scripts should also attempt to create problems that are known to occur with novice users. Examples of these would be security logon checks, breaks in data transmission, attempts to enter data out of the correct sequence or beyond the allowed time limits, entering alphabetical or numerical data in the wrong fields, and trying to change data once it has been finalized in the EPD.

It is important that the clinical team test from its own perspective without being influenced by the technical team's view. The technical team can support user acceptance testing by installing and qualifying the EPD system environment to be used for acceptance testing. The technical team should be on call to provide bug fixes and any other troubleshooting support needed to correct technical problems that arise during acceptance testing.

The exit criterion for this step is a test summary report written by the clinical team that describes the user test and records an official pass/fail status for the EPD. Following approval of a successful testing effort, the clinical-technical team prepares a combined go-live summary report for the trial. Management approval of that report signals that the electronic diary is considered validated, and the study is ready to launch.

Collaboration is key

The development and validation of a computerized EPD system begins with a clear protocol and set of specifications. In this article, we have shown how these specifications act as a road map for developing the software (SDLC) and the support structures (CTLC) for an EPD study. The clinical and technical teams have complementary responsibilities to develop the EPD system. The collaborative nature of the development process, as well as the expertise needed for many of the EPD development tasks, will be familiar to the clinical team. For those aspects of the development that are unique to an electronic diary system, the clinical team can rely on the technical team's technology and expertise.

We have shown how the system development and clinical trial life cycles collaborate and complement each other in bringing an electronic diary to life within the context of a clinical study. Documentation that emerges from the development process provides support for the notion that the EPD system is ready for release into a trial. The test summary report answers two important questions

• "Did we build the diary right to fit the design requirements?"

• "Did we build the right diary for subject use in this trial?"

The trial go-live summary report documents that

• the trial work process is ready to accept EPDs and train subjects in their use.

• the help desk is trained and ready to support launch of the study.

• the EPD development process is fully documented and ready for GCP and 21 CFR 11 audits.

• all sponsor and investigative participants have been trained and understand their EPD role going forward.

The fourth and final article in this series addresses the three remaining steps in the two life cycles: operate, maintain, and retire.

References

1. Unpublished data regarding frequency of diaries in clinical trials, DataEdge, 1999.

2. Michael R. Hufford & Alan L. Shields, "Electronic Diaries-Applications And What Works In The Field," Applied Clinical Trials, April 2002, 46-56.

3. Teri Stokes and Jean Paty, "Electronic Diaries, Part 1: What Is a Subject Diary, and How Do Regulations Apply?" Applied Clinical Trials, September 2002, 38-43.

4. Teri Stokes and Jean Paty, "Electronic Diaries, Part 2: The Role of the Clinical Protocol" Applied Clinical Trials, March 2003, 46-56.

5. Code of Federal Regulations, Title 21, Part 11, Section (U.S. Government Printing Office, Washington, DC).

6. Boehm and Pappacio, "Understanding and Controlling Software Costs", IEEE Transactions on Software Engineering, SE-14, no 10.

Teri Stokes, PhD, is director, GxP International, 131 Sudbury Road, Concord, MA 01742, (978) 287-4393, fax (978) 369-5620, email: drteristokes@gxpinternational.com, www.GXPInternational.com. Jean Paty, PhD, is one of the founders and chief quality officer of invivodata, inc., 2100 Wharton Street, Ste. 505, Pittsburgh, PA 15203, email: jean@invivodata.com, www.invivodata.com.