Are Event Rates and Effect Sizes Reported Accurately in Cardiovascular Trials?

Image Credit: © Tierney - stock.adobe.com

In the design of a randomized clinical trial (RCT), estimations for expected event rate and effect size are needed to accurately calculate sample size. In a study recently published on JAMA Network Open, the authors reviewed a select group of contemporary cardiovascular RCTs to find out just how accurately these values are being reported.¹

“During the design of a RCT, estimation of the expected event rate and effect size is a key component to calculating the sample size. Overly optimistic estimation of event rates and effect sizes may lead to underpowered trials,” the study authors wrote. “This study aimed to (1) evaluate event rate (control group) and effect size estimation accuracy in contemporary cardiovascular RCTs and (2) identify factors associated with higher accuracy of event rate and effect size estimation.”

The study utilized a systematic search of cardiovascular RCTs published in the New England Journal of Medicine, JAMA, and The Lancet between January 1, 2010, and December 31, 2019. The initial search returned 873 RCTs, with 374 fully reviewed and 30 subsequently excluded, resulting in 344 trials included in the analysis.

The results of the review returned inaccuracies in the reporting of expected event rates and effect sizes. The median observed event rate was 9.0% (IQR, 4.3% to 21.4%), which was significantly lower than the estimated event rate of 11.0% (IQR, 6.0% to 25.0%) with a median deviation of −12.3% (95% CI, −16.4% to −5.6%; P < .001). More than half of the trials (196 [61.1%]) overestimated the expected event rate. For effect sizes, the median observed in superiority trials was 0.91 (IQR, 0.74 to 0.99), which was significantly lower than the estimated effect size of 0.72 (IQR, 0.60 to 0.80), indicating a median overestimation of 23.1% (95% CI, 17.9% to 28.3%). Once again, overestimations were high with 216 trials (82.1%) overestimating the effect size.

“The main findings of this systematic review suggest that during the design of contemporary cardiovascular trials, event rates are often overestimated but event rate estimation accuracy is associated with significant refutation of the null hypothesis,” the authors said of the results. “We observed that 4 of 5 trials included in this review overestimated the effect size of the tested intervention. We identified several variables of trial design as independently associated with effect size estimation accuracy. The numbers of recruited participants did not significantly differ from the estimated sample sizes.”

According to the authors, trial investigators estimate these values from a sincere evaluation of data and their own expertise. This particular study did not identify the specific reasons for overestimation. However, possible factors may include the absence of robust data to inform event rate estimation, advancements in clinical care lowering event rates over time, and financial constraints or feasibility concerns prompting the restriction of sample size through optimal assumption of event rates and effect sizes.

Looking forward to future research, the authors suggest focusing on event types for specific diseases may be helpful in identifying areas where overestimation can be corrected.

“In this systematic review of contemporary cardiovascular RCTs, the event rates of the primary end point and effect sizes of an intervention were frequently overestimated,” the authors concluded. “This overestimation may have contributed to the inability to adequately test the trial hypothesis.”

Reference

1. Olivier CB, Struß L, Sünnen N, et al. Accuracy of Event Rate and Effect Size Estimation in Major Cardiovascular Trials: A Systematic Review. JAMA Netw Open. 2024;7(4):e248818. doi: 10.1001/jamanetworkopen.2024.8818