How to Transition RWE Studies from Non-Clinical to Regulatory-Grade

The Food and Drug Administration (FDA) has been working to enable greater adoption of real-world-evidence (RWE) in clinical and regulatory decisions, as mandated by the 21st Century Cures Act. The law has enabled a transition from using RWE for trial recruitment and marketing insight to using RWE for regulatory and reimbursement support. Within this transition, RWE is increasingly used to make clinical assertions. Thus, daily care will be adjusted based on what treatments are approved and which are reimbursed. As a consequence, expectations for completeness, accuracy, and validation of real world data (RWD) are increasing.  

The state of real world data

RWD available to the pharmaceutical industry includes EHR structured data, EHR unstructured data, patient registries, and claims and billing data. Claims data and registries constitute the large majority of RWD used in real world studies. However, the accuracy of RWD has repeatedly been proven to be low. For example, purchased claims and EHR structured data sets often have cohort accuracy between 30% and 70% percent. Even more commonly, accuracy is not checked at all. This may be sufficient for existing RWE use cases of marketing insight and trial recruitment as these approaches do not make clinical assertions in regulatory pathways and thus do not take into account data accuracy and generalizability sufficient for regulatory use. However, this low level of accuracy may not be sufficient for more advanced regulatory and reimbursement use cases that make a clinical assertion. For example, an assertion that one treatment is 10% better than another treatment would not be credible if the underlying data was 50% inaccurate. 

As the industry shifts toward using RWE to augment the standard of care, biotechnology and pharmaceutical firms are exploring how to run advanced studies to understand the real world impact of therapies on important clinical outcomes. Complementing randomized controlled trials with RWE can lower costs and provide compelling supplementary evidence in label expansion, post-marketing surveillance, and reimbursement. 

Achieving regulatory-grade real world data

There is an increasing discussion of the term “regulatory-grade.” We define “regulatory-grade” as data validity sufficient to support a clinical assertion. If two study arms are compared and one is 100% better than the other, it may be acceptable to have 50% data inaccuracy. Even if most of the missing data is skewed, the conclusion may still be valid. On the other hand, if two study arms are compared, and one is 10% to 20% better than the other, as is so often the case, then 50% data inaccuracy would be unacceptable. It would be expected that the data was 90%+ accurate, so that even if missing data was skewed toward sicker patients or a preferred outcome, the clinical assertion would still most likely be accurate. Given that RWE assertions are so often demonstrating 10% to 20% difference between study arms, most regulatory-grade RWE studies should be bolstered by data with 90%+ accuracy. 

The challenge is that when claims and EHR structured data are tested, they almost always show recall levels below 70% and too often have recall below 50%. Worse yet, there is a known skew. A doctor is far more likely to put a disease like a heart attack on the list if it is a bad heart attack. If a patient comes into the emergency department once and soon goes home, the chance that the disease does not end up on the problem list is far higher than if a patient is admitted to the intensive care unit and seen by a team of doctors multiple times. Therefore, the industry needs to get closer to the source data and enhance data based on the full record to achieve high accuracy.

To run highly accurate observational studies for subgroup analytics and comparative effectiveness, advanced data sources are needed, combined with advanced technology and expertise to increase RWD accuracy. The industry needs to consider new approaches to achieve required data validity to accomplish sufficient accuracy and to properly demonstrate study quality. Following are two steps necessary to produce a regulatory-grade RWE study.
 
EHR and underlying narrative data

A claims or EHR structured data set has limited information. If a disease, such as diabetic retinopathy, or a symptom, such as pain, is missing, and the underlying record is not available, there is not much that can be done to find these. Information cannot be created where it does not exist. In advanced use cases, it is becoming increasingly important to have access to the underlying data set.

By using advanced data sources, such as the patient narrative, and advanced technologies such as natural language processing (NLP) and artificial intelligence (AI), the underlying narrative data can be used to enhance the claims or EHR structured data. In this way, difficult inclusion criteria, exclusion criteria, and outcomes can be measured. 

Data validity studies

As RWD is increasingly used to make clinical assertions, changing the standard of care, an assumption that the data is accurate is not enough. Data validity should be assessed in all RWE studies that make clinical assertions.

It is fair to ask how data accuracy should be measured in claims and EHR data. Accuracy measurement requires a gold standard. While this seems simple, accuracy in data is hardly ever compared against a gold standard. The gold standard requires manual chart abstraction. Because people make mistakes, the chart abstraction should be completed by at least two clinicians. Then, interoperator consistency should be measured. The data set, whether claims, EHR structured, or EHR narrative, should be compared against the gold standard. This should result in a recall and precision score for each cohort. Recall and precision are similar to sensitivity and specificity, but more appropriate statistically due to the large number of diseases often tested. 

Today, the pharmaceutical industry is often limited to using RWE studies for trial recruitment and marketing. To enable a safe transition to using RWE to make credible clinical assertions, the industry is advised to consider new approaches to achieve regulatory-grade data. 

To run highly accurate studies that will influence the standard of care and present a solid case for regulatory and reimbursement support, biotechnology and pharmaceutical firms need to bring cohort accuracy up from the typical 30-70% to 90%+. If clinical assertions are to be made and the standard of care refined, these are the types of accuracy numbers required to assure that regulatory and reimbursement changes are made on good data rather than guesses.

Dan Riskin, M.D., M.B.A., is Founder and Chief Executive Officer of Verantos.