Optimizing Study Design in Real-World Evidence Generation: Rely on Input from a Specialist

Real-world evidence (RWE) is needed in addition to clinical trial data to understand drug effectiveness in a real-life setting and to profile patient populations in terms of their clinical characteristics and drug utilization. Studies designed to quantify the incidence and prevalence of health conditions, and/or population attributes such as risk factors are, in the scheme of clinical research, relatively straightforward studies to conduct. Clearly, they are not in the same league as randomized controlled trials (RCTs) in terms of their cost, operational complexity, timelines, or degree of risk. Consequently many sponsors assume that they can pursue these studies without input from an epidemiologist. This assumption can be risky, as epidemiologists understand how to minimize bias and ensure external validity-necessary elements of even these “simple” studies. 

Let’s look at a few of the most basic pharmacoepidemiological study designs and the associated risks of bias...

Cross-Sectional Studies: The Mainstay of Incidence/Prevalence Studies

Cross-Sectional Surveys (CSS) select participants based on the inclusion/exclusion criteria defined in the study protocol to profile a cohort of individuals at a single point in time. That is to say, there is no baseline established, no outcome measured, and no follow-up data collected. Rather, certain populations characteristics are identified that merely describe the cohort under study at a snapshot in time. CSS designs inherently lack an ability to examine the temporality of events e.g., cause and effect. 

Cross-sectional surveys are among the simplest and least expensive studies to conduct, and indeed, are very low on the hierarchy of evidence. (See Figure 1.) 

Medical Chart Reviews: Adding a Longitudinal Perspective 

Medical Chart Reviews (MCRs), in which data are drawn from patient’s medical charts (either manually in the case of physical charts or through a database of electronic medical records) can either be used to generate cross-sectional or longitudinal data. When these studies are conducted longitudinally, they may explore causality, providing insights beyond incidence and prevalence. At the same time, it must be acknowledged that medical charts are rarely complete for sponsors’ research purposes, given that they were, of course, created for a different purpose.[1]

Hybrid Studies: An Innovative Blend of CSS and MCR Designs

Hybrid CSS-MCR studies are a way of overcoming the limitations of one study design by adding another, complementary, study. Hybrid CSS-MCR studies can overcome the limitation of CSS studies with respect to being able to look at causality and outcomes since the patient history recorded in the medical chart can be used as the control, prior to exposure to a variable. And, hybrid studies can fill in gaps in the medical record by gathering additional details from patients, including on outcomes and adverse events. 

Hybrid CSS-MCR studies are conducted in the same way as individual CSS and MCR studies, but taken together, they allow for a much more comprehensive research investigation. 

The Caveats

Simple though they may be, CSS, MCR, and hybrid studies can produce robust findings that are generalizable to the population in the real world that will be using the treatment-in other words, they can have external validity. However, great care must be taken in the study design, and specifically the sampling strategy, to ensure generalizability. 

Below are some of the pitfalls that need to be guarded against:

· Non-Response Bias 

Cross-Sectional Surveys are frequent offenders of non-response bias, given the fact that patients who participate in research are different from those who do not. Admittedly, it is very difficult to eliminate all bias in CSS studies, but the goal must be to minimize it. 

· Selection Bias 

Selection bias can occur when the sampling is not well considered, such as in studies that deploy a convenience sampling strategy, rather than a probability sampling strategy (or make use of a representative data source). The difference between the two is marked. Probability sampling is achieved when every study participant has an equal chance of being selected for inclusion in the study. This ensures that the study population is representative of the target population since the selection occurs in a random fashion. Understanding representativeness is especially relevant for incidence and prevalence studies and requires an evaluation of how similar the sample population is to that of the population of interest by comparing certain characteristics (often demographic).[2]

Conversely, a non-probability sampling strategy does not enable the study population to have an equal probability of selection into the study, otherwise called non-random selection. CSS studies frequently rely on panels, clinician referrals, patient groups, traditional advertising, proprietary databases, and social media to generate evidence and in so doing are using non-probability sampling. Even if participants are selected from these sources at random, the question always comes back to whether or not the source was representative of the population of interest. Including a random selection process as part of a flawed sampling strategy will not ensure the generalizability of the study estimates. 

· Sample Size 

In order to test a study hypothesis, the study has to have sufficient “power.” In other words, the sample size has to be large enough to ensure that the study estimates are reliable. Conducting a study that is purely descriptive without considering the study size may result in study findings that cannot exclude the role which chance may have played. Having a sufficient study size is important to providing confidence that the study findings are not owing to sampling bias.  

· Confounding 

At study inception, potential confounders need to be identified so that data pertaining to these covariates can be collected at baseline, allowing the statistical modelling approach to control for them. If confounding is not assessed and addressed during the design and analyses stage, any resulting evidence of an association between two variables may be diluted or inflated due to the confounding. The range of the data collected in medical chart and EMR databases may sometimes make it challenging to control for all confounders. 

The Value of an Epidemiologist

As specialists in study design, epidemiologists know how to minimize (or avoid) the issues of selection bias, sampling bias, and underpowered studies. They will propose a study design and methodology for identifying, recruiting, and selecting participants and deliver robust generalizable data that provide insights into the target population.  

In working on a CSS, MCR, or hybrid study, an epidemiologist will:

· Consider the sampling strategy up front as part of the study design 

· Explore real-world data sources and consider those that are fit for the research question 

· Identify a data source that enables the generalizability of the sample population 

· Determine the most appropriate sampling strategy 

· Work with biostatisticians to calculate the appropriate sample size 

· Ensure that any study limitations are minimized and clearly explained 

Unfortunately, if a study design was poorly conceived and allowed for bias, the flaw may not become apparent until after the data are collected and the analysis is underway. At that point, there’s very little that can be done to salvage it. Sampling bias, for example, may be discovered when the prevalence measured by the study is significantly higher or lower than what is expected (based on findings from other studies). To prevent undermining the study integrity in this way, it is advisable to involve an epidemiologist at the study inception or at the very least to have the study protocol critiqued by an epidemiologist. With their very specialized focus on study design, epidemiologists can ensure the research design is fit for the research objectives and ensure robust and reliable downstream study findings. 

Rob Sambrook, Divisional Principal, ICON. He can be reached at Robert.sambrook@iconplc.com

[1]Gearing RE, Mian IA, Barger J et al. A Methodology for Conducting Retrospective Chart Review Research in Child and Adolescent Psychiatry, Journal of the Canadian Academy of Child and Adolescent Psychiatry, 2006; 15 (3); 126-34.

[2]Safdar, N, Abbo LM, Knoblock MJ et al. Research Methods in Healthcare Epidemiology; Survey and Qualitative Research, Infection control and hospital epidemiology 2016;37(11):1272-77.