Are Event Rates and Effect Sizes Reported Accurately in Cardiovascular Trials?
Systematic review of over 300 randomized clinical trials sought to determine how often these values were overestimated in contemporary cardiovascular trials.
In the design of a randomized clinical trial (RCT), estimations for expected event rate and effect size are needed to accurately calculate sample size. In a study recently published on JAMA Network Open, the authors reviewed a select group of contemporary cardiovascular RCTs to find out just how accurately these values are being reported.1
“During the design of a RCT, estimation of the expected event rate and effect size is a key component to calculating the sample size. Overly optimistic estimation of event rates and effect sizes may lead to underpowered trials,” the study authors wrote. “This study aimed to (1) evaluate event rate (control group) and effect size estimation accuracy in contemporary cardiovascular RCTs and (2) identify factors associated with higher accuracy of event rate and effect size estimation.”
The study utilized a systematic search of cardiovascular RCTs published in the New England Journal of Medicine, JAMA, and The Lancet between January 1, 2010, and December 31, 2019. The initial search returned 873 RCTs, with 374 fully reviewed and 30 subsequently excluded, resulting in 344 trials included in the analysis.
The results of the review returned inaccuracies in the reporting of expected event rates and effect sizes. The median observed event rate was 9.0% (IQR, 4.3% to 21.4%), which was significantly lower than the estimated event rate of 11.0% (IQR, 6.0% to 25.0%) with a median deviation of −12.3% (95% CI, −16.4% to −5.6%; P < .001). More than half of the trials (196 [61.1%]) overestimated the expected event rate. For effect sizes, the median observed in superiority trials was 0.91 (IQR, 0.74 to 0.99), which was significantly lower than the estimated effect size of 0.72 (IQR, 0.60 to 0.80), indicating a median overestimation of 23.1% (95% CI, 17.9% to 28.3%). Once again, overestimations were high with 216 trials (82.1%) overestimating the effect size.
“The main findings of this systematic review suggest that during the design of contemporary cardiovascular trials, event rates are often overestimated but event rate estimation accuracy is associated with significant refutation of the null hypothesis,” the authors said of the results. “We observed that 4 of 5 trials included in this review overestimated the effect size of the tested intervention. We identified several variables of trial design as independently associated with effect size estimation accuracy. The numbers of recruited participants did not significantly differ from the estimated sample sizes.”
According to the authors, trial investigators estimate these values from a sincere evaluation of data and their own expertise. This particular study did not identify the specific reasons for overestimation. However, possible factors may include the absence of robust data to inform event rate estimation, advancements in clinical care lowering event rates over time, and financial constraints or feasibility concerns prompting the restriction of sample size through optimal assumption of event rates and effect sizes.
Looking forward to future research, the authors suggest focusing on event types for specific diseases may be helpful in identifying areas where overestimation can be corrected.
“In this systematic review of contemporary cardiovascular RCTs, the event rates of the primary end point and effect sizes of an intervention were frequently overestimated,” the authors concluded. “This overestimation may have contributed to the inability to adequately test the trial hypothesis.”
Reference
1. Olivier CB, Struß L, Sünnen N, et al. Accuracy of Event Rate and Effect Size Estimation in Major Cardiovascular Trials: A Systematic Review. JAMA Netw Open. 2024;7(4):e248818. doi: 10.1001/jamanetworkopen.2024.8818
Including Women of Childbearing Age in Clinical Research
March 26th 2024In recognition of International Women's Month, we're featuring this recent talk between Associate Editor Miranda Schmalfuhs and Marie Teil, Global Head of UCB’s Women of Childbearing Age Program. They speak about the specific challenges women with chronic illnesses face when accessing appropriate treatment and participating in clinical trials, UCB's Women of Childbearing Age Program and it’s most successful strategies, and much more.
Improving Engagement While Maintaining Data Integrity & Validity
March 19th 2024In recognition of Women's Health Month, we're featuring this recent talk between Associate Editor Miranda Schmalfuhs and uMotif's Chief Product Officer, Julia Lakeland, discuss new technologies improving patient engagement and reducing the emotional and logistical burdens of participation, ethical considerations that should be addressed when implementing those technologies, while ensuring patient privacy, and much more.
