Barriers and Solutions to Smart Clinical Program Designs

Due to ever increasing expenditures and difficulties associated with successful development and launch of innovative new medical products, the pharmaceutical industry has devoted substantial effort to root cause analyses.^1,2,3 Taken together, these analyses show increasing complexity of clinical development with concomitant significant risks to the return on R&D Development investment.⁴ Furthermore, efforts to streamline clinical development processes with automation and digitization of data capture and reporting have failed to improve overall product approval rates or lower the cost of clinical research.⁵ The seeming paradox of process improvement without concomitant positive impacts on clinical development outcome led to the formation of a collaborative team in 2013 within the European Federation of Pharmaceutical Industries and Associations (EFPIA) called “Smart Program Design.” The Smart Program Design team published an article in 2015, “Smart Program Design through a Common Information Model,” which proposed improvements to the standard clinical program design approach involving precompetitive collaboration and information sharing.⁶ The group identified four key challenges to consistent and repeated smart clinical program designs:

• Design information is captured ad hoc

• There is an inability to learn from past programs, both within organizations and externally

• Current industry information standards do not cover program level or the rationale behind these designs

• There is limited opportunity for progress in future opportunities in improving clinical program⁶

Thus, a gap exists for a common information model to describe the key building blocks essential in representing a clinical program and the corresponding design rationale, which allows for organization of information for reuse, facilitation of communication, and enablement of innovation. The common model should bridge between data and decision, and between decision and target product profile (TPP).⁶

A presentation of these concepts at the 2014 Clinical Data Interchange Standards Consortium (CDISC) IntraChange conference resulted in a recommendation by Wayne Kubick, Chief Technical Officer (CTO) of Health Level Seven International (HL7) and former CTO of CDISC, to initiate work on a broader set of information standards to address clinical development program design, including the capture of design decisions, rationale, and references. 

Current data standards focus on describing and capturing the clinical trial protocol and data rather than the design process. No data standards that apply to research operations or their related critical decision-making process exist. Hence, we took advantage of an opportunity in the Pharmaceutical Users Software Exchange (PhUSE) Semantic Technology group to develop the Clinical Development Design (CDD) Framework.

The purpose of the CDD Framework is to apply the principles and process of a design-based approach to the development program for a medical product, with emphasis on the identification, collection, and use of relevant information in a structured manner. This will enable decision-making and construct a learning system framework that facilitates ongoing improvement and efficiency. The model is based on the principle that decision-making is dependent on and enhanced by what is termed “enabling information.” In this model, the capture and use of enabling information is based on design principles. 

Overview of design decisions

Our first question was how are design decisions made in other industries? Other domains with ongoing dynamic decision-making can inform clinical research program development. In Thinking: Fast and Slow, Daniel Kahneman explores decision-making.⁷ He stresses that each industry needs ways to ensure the product’s quality from initial design through production and final inspection. In each stage, we are framing the problems that need to be solved and the corresponding decisions that need to be made. He suggests that at each of these stages-design, production, and final inspection-we look for ways to improve our design decisions in order to improve our product. Kahneman also stresses the concept of “noise” or lack of reproducibility, the variability in design decisions.⁸ He recommends ways to decrease the noise in design decisions by building algorithms, rather than depending solely on individual human judgment.

Overcoming human inconsistency as a solution to better decision making is supported by the work of Theresa Winhusen.⁹ She focuses on the pre-implementation phase of multi-site pharmacological clinical trials and details the causes of delays and how they affect the study. The three primary causes for delays in the pre-implementation period are: (1) unforeseeable events, (2) underestimation of how long a project will take, and (3) difficulties in coordination of the many parties involved in the clinical trial. She proposes that with the proper tools, the impact of these types of delays can be overcome.

Roger Buehler, consistent with Winhusen and Kahneman, shows examples of planning fallacies found in a wide range of industries, and presents tools that can assist in more accurately planning a project.¹⁰

While incorporating algorithms and tools into the decision-making process is advantageous to improve consistency and precision, the tools must be applied to sufficient and accurate information to be effective. K.M. Sutcliffe notes that too little as well as too much information can be difficult for a manager to interpret and apply to successfully manage a project.¹¹ Dan Lovallo and Kahneman show how managers need the subjectivity removed from the design decisions to put out a more accurate estimation of the success of the product.¹² Kahneman suggests managers and decision-makers can reduce noise and improve consistency by employing filters.⁸ Sheehan, Hirschfeld, Foster et al. point out that another way to reduce the noise is to use a set of common data elements (CDE) to obtain a more accurate signal.¹³

Describing design decisions in CDD

High quality design is most commonly manifested by success in the creation, development, commercialization, and optimization of a tangible product with a specified use, such as a car, a house, or a software application. While product users make multiple decisions during its use, the product is typically not changed on a daily basis. Design in the realm of clinical research is different, however, in that the research can be reevaluated and adjusted continuously. Clinical research on a medical product is dependent on the process of creation, development, and optimization of a scientifically robust, operationally feasible, and economically relevant research program to generate compelling evidence for internal decision-making, regulatory evaluation, and public confidence in its efficacy, safety, and cost-effectiveness. In contrast to the mass manufacture of consumer products, each clinical development program for every biomedical product is unique.

Design decisions in clinical development programs and trials rely on consolidating, analyzing, weighing, and prioritizing a broad range of sources of information from guidance, past experience, expert advice, and real-world evidence. This information is typically captured in an unstructured format, in the form of clinical development plans, protocols, documents, and presentations.

Such a process is not optimal. Designing a successful clinical research program can benefit from:

• A robust and comprehensible framework

• Collaboration across many domains

• Generation, capture, and effective sharing of data and historical information

• Relevant tools

The CDD Framework we propose encompasses the support for design-based decisions and how to apply them to a clinical development program. It should accurately tell the story of how plans, data, and information evolve during the full life cycle of a therapeutic product from before clinical testing, then entry into first-in-human dosing, followed by the clinical trials, marketing authorization, as well as the experience from postmarketing surveillance and subsequent studies. This proposed CDD Framework is different from a Clinical Development Plan (CDP), which we define as the tactical execution of a research plan.

Mapping the CDD Framework

The CDD Framework provides a methodology for the organization and preservation of information critical to medical product development decisions, information we refer to as “enabling information” (EI). EI includes process data, timelines, costs, and resource burden, among other quantitative and qualitative items. EI exists at every phase of the process, from concept to non-clinical and clinical testing, through regulatory approval, marketing, and post-marketing surveillance. The complete EI is ideally a comprehensive representation of an entire medical product development program. 

Thus, the CDD Framework collates the EI and captures the perspectives of all stakeholders, scientists, administrators, care providers, patients, regulatory agencies, financial stakeholders, collaborators, and sponsors in the product development process. One approach to collating EI and populating the framework is to focus on three well-recognized, general objectives of medical product development (see Figure 1):¹⁴

• Set goals that describe the product and patient population who will receive the product 

• Characterize the product administration (intervention) and define outcome measures (both efficacy and safety)

• Assess product viability against a benefit and risk profile

These three well-recognized objectives of product development allow for a tentative mapping of our proposed CDD Framework. The following is a suggested approach to this mapping.

Goals and target population

When setting the goals, information on the condition (indication), the target population, and the expected benefit to that population (see Figure 2) is needed. This step also requires a description of the product under development, as well as measurable endpoints that are connected to the objectives and measures. The target population is typically defined by phenotype and demographics, but one can also consider genotype, geographic location, age, or developmental stage and lifestyle. Information generation and collection in this step may originate from many sources, including scientific and medical subject matter expertise, epidemiology, outcomes research, biostatistics, real-world data, and clinical operations.

Intervention and outcome measures

Characterizing the intervention requires investigation of its expected clinical and biological effects, and establishing a correlation between exposure and effect. These tasks will also require input from scientific and medical subject matter experts and statisticians. If relevant, a central role should be taken by experts in biomarker selection and analysis. This step requires an understanding of the nature of the intervention, dosing, and administration. Route, frequency, duration, and other parameters related to exposure need to be understood and documented. There needs to be a focus on defining informative and relevant outcome measures that vary with deterioration or amelioration of the clinical condition in a predictable manner. The clinical trial assessments must be feasible, acceptable, linked by time, and connected to the outcome measures.

Risk and benefit evaluation

There needs to be an emphasis on risk management (see below), with assessments and outcome measures demonstrating an acceptable risk/benefit profile. This profile must be based on clinical trials with credible study design maximizing participant safety while minimizing bias and uncertainty as well as analysis of data assured of integrity. 

Validation of design output

To determine that all aspects are covered in the design, the perspectives of patients, investigators conducting the trial, oversight bodies, regulators, and the sponsor should be reviewed. Table 1 gives examples of some of the aspects to be validated as the output of the design of a clinical trial protocol within the CDD Framework.

To summarize, the vast number of questions and ensuing answers required for each objective are interdependent and temporally linked. With that in mind, the design of the CDD Framework requires a comprehensive understanding of the design decisions and why these decisions were made. Each objective is supported by enabling data, consisting of regulatory information, early research and development data, clinical study data, epidemiological data, payer-focused data, and justifications.

The clinical design effort will not necessarily be a process in which goals are set, interventions characterized, and risk-benefit demonstrated in a linear fashion. As new information is generated that more clearly demonstrates aspects of the benefit-risk profile, the intervention, the condition, or the target population may become better characterized, leading to the setting of new goals. Also, at any given stage of medical product development or the life cycle management of a medical product, there will be key questions that remain unanswered, or the uncertainty about the outcomes based on the collected evidence may become greater than desired. The CDD Framework captures these gaps and uncertainties, and helps moving forward by prompting the establishment of a logical plan for setting goals that will fill those gaps and decrease uncertainties. 

The CDD Framework is intended to evolve into a knowledge base for the design of products supporting key design objectives, and to facilitate the validation of those objectives. It should properly represent the interdependencies between design constraints, decisions, and information-and facilitate the communication of those relationships between stakeholders to ensure that the work designated in the CDD actually takes place.

Risk management

There are many types of risks to be addressed throughout the CDD Framework, each of which requires appropriate management. Besides risks relating to regulatory issues, risk considerations within CDD include those for business and scientific validity, as well as concerns such as loss of time, opportunity, effort, and reputation.

Certain risks will not be addressed in the CDD Framework, because those risks could not reasonably be attributable to aspects of design, such as reporting delays, data issues, misconduct identified, incorrectly enrolled patients at a site, etc. Moreover, the European Medical Association (EMA) notes that “risk management is a systematic process for the assessment, control, communication, and review of risks associated with the planning and conduct of clinical trials and clinical development programs.”¹⁵ The CDD Framework should help to identify risks to determine what can happen, when, where, how, and why-and to enable analysis of the likelihood of the occurrence and detection. Finally, there should be decisions on acceptable tolerance levels.

Tools – design examples

Implementation of the CDD Framework can leverage many of the tools, methods, and technologies that have been applied to other areas of knowledge acquisition, storage, and analysis. Utilization of various combinations of these tools can enhance the power and interoperability of the CDD Framework for the capture of enabling information, decision options and rationale, and relationships of choices to outcomes.

Semantic technology

The conventional approach to store and access information is by categorizing and indexing information into relational databases based on static structures. XML-based data standards, such as those currently used in CDISC data capture and transfer standards, are designed for such structured data. A primary challenge here is that information associated with design thinking can initially be apparently unstructured, incomplete, or subject to further changes and definitions. However, similar to data standards, updates are inevitable.

Semantic technologies represent a family of technologies related to the capture, storage, indexing, querying, and classification of unstructured data governed by the World Wide Web Consortium (W3C).¹⁶ In particular, semantic technology allows the development of flexible information models, where concepts can have different meanings, links can be created between disparate information sources, and information is incomplete. 

For semantic technology, well-defined rules are established that convey meaning to specific words.¹⁷ In doing so, an assortment of words that have the same meaning (as defined by semantic rules) allows for an overarching search with far greater returns than relying on a search for each individual word. Moreover, with linked web data, the user can search for information on the Internet and circumvent the need to rely entirely on conventional relational databases. However, some of the linked data sources can be:

• Unstable

• Outdated/not maintained

• Difficult to discover, explore

• Largely undocumented

• Time-consuming to establish

The flexibility offered by semantic technology makes it well suited to be leveraged for the CDD information model. We need to obtain relevant information for input into the program design, as well as the intended protocols, appropriate data analyses, clinical study reports, and regulatory submission. These components, attainable through semantic technology, are incorporated in the CDD Framework in a comprehensive yet temporal arrangement.

Ontology

For any information model, a common vocabulary or ontology is a prerequisite. Ontologies help us to organize terms and define relationships, to enable reuse of domain knowledge and separate domain knowledge from the operational knowledge.¹⁸ Ontologies organize domain knowledge in a common structure of entities and their relationships.¹⁹

An example of applying ontologies for semantic data supporting clinical research is the ontology for FDA regulations in the Resource Description Framework (RDF), being created by the PhUSE Reg2RDF group.²⁰ Currently, the work focuses on indexing 21 CFR terms and evaluating the key phrases and presenting a web interface. The best way to organize the review process is part of their development process. The CDD Framework aims to apply a similar approach for establishing a corresponding ontology.

In order to develop the CDD ontology and to begin mapping the design process, we can start with the checklists from the FDA (including the TPP and prescribing information labeling templates and checklists) and quality by design reference tools from the Clinical Trials Transformation Initiative (CTTI) as well as other data sources.^21,22,23

Before we can apply the terms, we must follow the examples found in developing biomedically-based common data elements (CDE).¹³ We need to bring together all of the above terms in a single resource with links, redundancies, and hierarchical relationships. Our starting place in building our ontology is Vanderbilt University’s Research Electronic Data Capture (REDCap) application.²⁴ This application allows us to capture redundancies and provide links between the terms that we are referencing.

Cmap – concept mapping tools

A semantic information model can serve as the backbone for software applications. However, a visual representation of the CDD Framework can also serve as a guide for design teams on its own. A two-dimensional open source concept mapping tool is available from the Florida Institute for Human & Machine Cognition (IHMC).²⁵ Using this software, we will be able “construct, navigate, share, and criticize” our models of the CDD Framework. Cmap tools allow us to not only share our concepts and understanding of the CDD Framework, but to link our maps to related concept maps and other types of media.26,27

Figure 3 below (click to enlarge) maps the FDA’s 21 CFR Part 201 regulations with the Cmap tool. We can then hyperlink and download each of these sections of the regulations. For the CDD Framework, we can develop Cmaps for each of our areas: setting goals, characterizing the intervention and delivering the medical product, and link the information that medical/scientific, operations, and regulatory groups need for their work.

Visual interactive Information model

The Cmap concept maps give us a two-dimensional mapping. The next step beyond the two-dimensional concept maps is the work done by Kerstin Forsberg and Maria Benjegard with Neo4j mapping.²⁸ This tool allows for multi-dimensional modeling and linking.

Figure 4 depicts the concept of a visual, interactive information model representing the interdependencies between activities, information, and decisions.

Figure 5 (click to enlarge) is an example what is considered for “Evaluation of Standards” in Benjegard and Forsberg’s Understanding Trial Design.²⁸ Each activity contains:

• What should be considered

• When it needs to happen, influencing work to come

• Where to learn more (e.g., guides to formal procedure and guidance)

The medical product development data and metadata entails collection of the required factors describing not only what, where, and when a decision was made by one or all of the stakeholders, but also what alternative decisions were proposed, why these were proposed, and why these decisions were accepted or rejected. In order to support the development of reproducible design-based decisions, the data and metadata for supporting decisions must be detailed and accessible. The value of the CDD Framework is dependent on easy application of this information model. We envision that it will facilitate robust design decision support, including toolsets. However, an interactive visual model can also directly contribute to the understanding and representation of the process.

Systematic collation of EI depends on application of CDE, collection and defining of easily retrievable data, and conforming to definitions that are agreed to by design stakeholders.¹³ Through the identification of enabling information, the components of clinical design may be assembled into a framework that optimizes a particular use case and establishes the possible reuse of and learning from the decision-making process. To maximize the utility of enabling information, it should be collected with fidelity, quality, stringency, and timeliness.

Regulatory Interactions

A clinical development program framework exists within an environment under the oversight of regulatory agencies. A general hierarchy regarding authority and legal enforceability is as follows:

• Statutes are binding and generally describe principles and goals to achieve an outcome such as assurance that products intended for human medicinal use are safe and effective. In the US, they are developed by the legislative branch and signed by the executive branch. They are binding and legally enforceable.

• Regulations are based on statutes and generally provide further details on implementing the intent of the statutes. Regulations are developed by the executive branch. They are binding and legally enforceable.

• Guidance documents are developed by individual agencies and represent a default recommendation for applying laws and regulations to particular topics. Guidance documents are usually not binding; however, variance from agency recommendations usually is expected to be supported by a scientific, logistical, or other rationale.

• Specific agreements between a party and an agency, such as a special protocol assessment, are on a case-by-case basis and considered binding.

In the CDD Framework, the interactions between data, scientific principles, regulatory principles, business decisions, and value decisions can be captured, analyzed, displayed, and used to inform future decision-making.

A regulatory agency as a partner for input and agreement on any product development plan is an important expectation. The CDD Framework for a particular medical product or planned indication can offer valuable insight and allow knowledge-sharing about the product.

The overall design of a product development program from inception to postmarketing success is important for regulatory review, because every development program activity needs to be compliant with the regulatory environment(s) and expectations in which the medical product is intended to be studied and marketed. The CDD Framework can provide a blueprint for a regulatory review team to valuate a proposed program and may suggest metrics for comparison with programs for the same class of medical products for the same condition or for similar populations. These will help identify potential remedies to address deficiencies.

Integrating a target product profile and the CDD

A target product profile (TPP) is a format for a summary of a drug development program described in terms of labeling concepts that may be very helpful in its program design.²¹ While its submission to a regulatory agency is voluntary, a pharmaceutical sponsor may share a TPP with the agency to facilitate communication regarding the design for clinical development.

The FDA has provided a guidance document detailing the labeling concepts in TPP and suggestions for what should be addressed in the label.²¹ Sponsors often want more information on adequately addressing these concepts during clinical development.  More information can be found in checklists from the FDA, such as the “Selected Requirements of Prescribing Information” (SRPI) checklist, which details 41 required and optional items for the labeling of drugs and biologics.²² When fully developed, our CDD Framework may offer us a link between the TPP and these checklists. Other potential advantages could be the linking of the TPP with quality by design and risk assessment tools.^23,29

When using the TPP, the CDD Framework needs to include details about these labeling concepts: indications and usage, dosage and administration, dosage forms and strengths, contraindications, warnings and precautions, adverse reactions, drug interactions, use in specific populations, drug abuse and dependence, overdosage, description, clinical pharmacology, nonclinical toxicology, clinical studies, references, how supplied/storage and handling, and patient counseling information. Recommendations to address these concepts can be found in the FDA guidance document on the TPP.²¹ As discussed above, one may be able to link recommendation in the TPP guidance to the SRPI checklist to ensure compliance with the labeling regulations.

Discussion

With the increasing challenges associated with successfully completing drug development, which includes issues about attending to medical product launch and postmarketing surveillance, contract research organizations (CROs) and pharmaceutical companies have started to explore ways to improve clinical development design decision-making. Automation has aided the execution, management, and reporting of clinical trials, but this can only contribute partially to efficiency. The clinical research community has realized that expert scientific and regulatory guidance, information enabling decision-making, and retention of project memory are necessary for successfully completing drug development.

A design paradigm can be summed up by four key steps: “Define – Integrate – Prototype – Crystallize.”³⁰ Prototyping and crystallizing would require the integration of accessible evidence, which may include real-world evidence. The process starts with defining and framing the problem questions in order to identify clear and appropriate objectives. Access to historical actual evidence and data are needed. Tools are also needed to enable calculations, visualization of design concepts, and engagement of all the stakeholders through clearly defined and stated goals.

Technologies that easily link design components and decisions to the information/data are needed in clinical development to overcome the unavoidable loss of historical knowledge. Even with existing technologies that allow for the creation of relationships between design components and the capture of “decision points,” one can only capture the final decision, the rationale, and/or supporting data in text-based minutes of meetings. There is often the intent to transcribe the information from documents such as meeting minutes to a better system, but competing needs generally prevail regarding resource allocation. Thus, the technology must be easy to use and not time-consuming. It must be intuitive so that teams can capture the relevant information in real-time during the design meetings. The teams must be tasked with not only capturing information around decisions implemented in the design, but also what was not implemented, thus preserving design alternatives for future consideration. The following are issues relating to the CDD Framework which organizations need to consider:

• What are the key decisions to be made during a product development life cycle?

• How is the enabling information captured, archived, and analyzed?

• How is the enabling information made available for review of the ongoing project and for future projects?

• How can information be captured in real-time?

• Is there something we can decide that does not need documentation? If so, how is that decision reached?

• How do we document the decision not taken?

• Reproducible design decisions-given the same information, will we always come to the same conclusion? What other factors influence decision-making?

We embark on an endeavor that, as noted by Kahneman, is spreading across all industries to reduce “noise” and improve decision making.⁹ Our goal is to develop a framework and a knowledge base of linked data on the CDD Framework that will serve as an information model for clinical research stakeholders, including considerations for regulatory input and risk assessment/management. This is a first step in providing collaborative tools for CDD-based decision-making and establishing the validity and applicability of the model. 

We are currently working on adapting tools, including a CDD ontology, to be used to map and implement the design and decision-making process.

Mary Banach, PhD, MPH, Department of Biostatistics, Vanderbilt School of Medicine; Hon-Sum Ko, MD, Division of Dermatology and Dental Products, Center for Drug Evaluation and Research (CDER); Steven Hirschfeld, MD, PhD, National Institute on Deafness and other Communication Disorders, The Uniformed Services University for the Health Sciences, Public Health Service; Maria Benjegård, MSc, AstraZeneca Global Medicine Development, Biometrics and Information Science; Ian Fisher, MChem, QuintilesIMS; Mitra Rocca, Dipl. Inform. Med., Office of Translational Sciences, CDER;Rashedul Hasan, PhD, Office of Translational Sciences, CDER; Kerstin Forsberg, Ph Lic, Informatics, AstraZeneca Global Medicine Development, Biometrics and Information Science; Dale Plummer, BS, Department of Biostatistics, Vanderbilt School of Medicine; Courtland E. Yockey, MS Biochemistry, R&D Information, AstraZeneca Pharmaceuticals; Johann Proeve, PhD, Clinical Data Management Consulting; Laszlo Vasko, MS, Information Science, MS, Clinical Research Operations Management, R&D Information, AstraZeneca Pharmaceuticals

* The opinions expressed in this article are those of the authors and do not necessarily represent the opinions of respective companies or organizations, or regulatory authorities. The content should not be interpreted as a data standard and/or information required by regulatory authorities.

* Acknowledgement: This paper and project grew out of our collaborations in the PhUSE Semantic Technology Working Group. This paper could not have come to fruition without the boundless support of Tarek Elbeik, CEO of Elbeik Associates, in gathering all of our diverse materials and perspectives together in one succinct outline; Asiyah Yu Lin, Research Fellow at the FDA, for providing us a primer on building ontologies; and Matt Austin, Director, Development Data & Analytics, Development Center at Amgen, in grounding all of our original discussions in developing a functional model. Finally, we would like to acknowledge Cara Willoughby, Principal Scientific Advisor of Bringing Design Thinking to the Biopharma Industry, QuintilesIMS; and Marc Buyse, Founder of the International Drug Development Institute and CluePoints, for their reviews of this article. Marc’s insightful reviews have already become the basis for our next two papers.

References

1. DiMasi JA, Grabowski HG, Hansen RW. Innovation in the pharmaceutical industry: new estimates of R&D costs. Journal of Health Economics 2016;47:20-33.

2. Getz KA, Campo RA, Kaitin KI. Variability in protocol design complexity by phase and therapeutic area. Drug Information Journal 2011;45:413-20.

3. Hay M, Thomas DW, Craighead JL, Economides C, Rosenthal J. Clinical development success rates for investigational drugs. Nature Biotechnology 2014;32:40-51.

4. 2012 CMR International Pharmaceutical R&D Factbook. PDF2012 June 2012 http://cmr.clarivate.com/pdf/2012-cmr-factbook-exc_cbr-en.pdf

5. Getz KA, Wenger J, Campo RA, Seguine ES, Kaitin KI. Assessing the impact of protocol design changes on clinical trial performance. American Journal of Therapeutics 2008;15:450-7.

6. Vasko L, Sundgren M, Bachmann P, et al. Smart Program Design Through a Common Information Model. Therapeutic Innovation & Regulatory Science 2015;49:116-25.

7. Kahneman D. Thinking, Fast and Slow. Penguin Books Limited; 2011.

8. Kahneman D, Rosenfield AM, Gandhi L, Blaser T. NOISE: How to overcome the high, hidden cost of inconsistent decision making. Harvard Business Review 2016;94:38-46.

9. Winhusen T. Not getting lost in translational science: A tool for navigating the pre-implementation phase of multi-site pharmacological clinical trials. Applied Clinical Trials 2014;23:36-9.

10. Buehler R, Griffin D, Ross M. Exploring the" planning fallacy": Why people underestimate their task completion times. Journal of Personality and Social Psychology 1994;67:366.

11. Sutcliffe KM, Weber K. The high cost of accurate knowledge. Harvard Business Review 2003;81:74-82, 129.

12. Lovallo D, Kahneman D. Delusions of success. Harvard Business Review 2003;81:56-63.

13. Sheehan J, Hirschfeld S, Foster E, et al. Improving the value of clinical research through the use of Common Data Elements. Clinical Trials (London, England) 2016;13:671.

14. Hirschfeld S. (2016). Clinical Trial Design Process: An Introduction. Paper presented at the PhUSE - FDA Computational Science Symposium, Silver Spring, MD. http://www.phusewiki.org/wiki/images/f/fb/2016_Hirschfeld_CSS_Development_Plan_Design.pdf

15. Proeve J. Clinical Study Risk Management: What Does it Really Mean & How do You Do It?  DIA Annual Meeting; 2014; San Diego, CA.

16. Semantics C. Semantic Web vs. Semantic Technologies. 2016. https://supportcenter.cambridgesemantics.com/semantic-university/semantic-web-vs-semantic-technologies

17. Williams T, Andersen M. (2016). Semantics 101 for Pharma. Paper presented at the PhUSE-FDA Computational Science Symposium, Silver Spring, MD. http://www.phusewiki.org/wiki/index.php?title=Semantic_Technology_Curriculum

18. Noy N, McGuinness D. Ontology development 101: A guide to creating your first ontology; 2001.

19. Yu Lin A. Three Ws of Ontology; 2015 http://www.phusewiki.org/wiki/index.php?title=Representing_ClinicalProgramDesign_in_RDF

20. Yu Lin A, Hasan R, Ko H, Banach M, Ayalew K, Rocca M. Making 21CFR a linkable resource for Semantic Web Applications; 2016 http://www.phusewiki.org/docs/CSS%202016%20Posters/PP17_Final.pdf

21. FDA. (2007). Guidance for Industry and Review Staff: Target Product Profile - A Strategic Development Process Tool. Washington, DC: HHS FDA http://www.fda.gov/downloads/drugs/guidancecomplianceregulatoryinformation/guidances/ucm080593.pdf

22. FDA. (2009). Selected Requirements of Prescribing Information, 21CFR201.56.  http://www.fda.gov/downloads/drugs/guidancecomplianceregulatoryinformation/lawsactsandrules/ucm373025.pdf

23. Clinical Trials Transformation Initiative (CTTI). QbD (Quality By Design) Toolkit. 2016. http://www.ctti-clinicaltrials.org/toolkit/qbd

24. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)-a metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics 2009;42:377-81.

25. IHMC. Cmap Concept Maps. 2014. http://cmap.ihmc.us/

26. Cañas AJ, Hill G, Carff R, et al. CmapTools: A knowledge modeling and sharing environment.  Concept maps: Theory, methodology, technology Proceedings of the first international conference on concept mapping; 2004. p. 125-33.

27. Novak JD, Cañas AJ. The Theory Underlying Concept Maps and How to Construct and Use Them: Florida Institute for Human and Machine Cognition; 2008 http://cmap.ihmc.us/docs/pdf/TheoryUnderlyingConceptMaps.pdf

28. Benjegard M, Forsberg K. Understanding Trial Design using Visualization. Paper presented at the PhUSE Annual Conference, Vienna; 2015 http://www.phusewiki.org/docs/Conference%202015%20TT%20Presentations/TT06_Phuse%202015%20Understanding%20Trial%20Design%20using%20Visualization%20Monday%2012th%20October_version%202.pdf

29. CTTI. CTTI Recommendations: Quality by Design; 2015. http://www.ctti-clinicaltrials.org/files/QbD_toolkit/CTTI%20Quality%20by%20Design%20Recommendations_FINAL_1JUN15.pdf

30. Sax F. Structured Design Methodologies In Clinical Research. Contract Pharma; 2013.