FDA-Approved Cancer Drug Trials May Overestimate Effects.
The Food and Drug Administration’s decision to approve a new cancer drug is based on pivotal trials that typically focus on whether results are statistically significant. But a new study led by School of Public Health researchers finds that relying on statistical significance can lead to overestimating the effects of a drug.
The study, published in the Journal of Clinical Epidemiology, found that, while there was almost no risk of falsely claiming a new drug was beneficial, smaller trials overestimated the magnitude of the true treatment effect by as much as 30 percent.
“Effects seen through the significance lens can appear larger than they truly are,” says lead study author Emily Lord (SPH’17), who was a student during the study. “The overestimation leads to potentially skewed understanding of the drug effect, and affects the design of subsequent replication trials.”
The researchers used two novel metrics to scrutinize 42 randomized controlled trials supporting 39 approved cancer drugs. The first metric was an estimate of how likely it was for a trial to falsely claim that a drug was beneficial, and the second metric was to quantify how much estimated effects differed from true effects.
Focusing on the 38 randomized controlled trials that reported an increase in overall survival for cancer patients, the researchers found that there was effectively no risk of falsely claiming a drug was beneficial. However, the researchers found that smaller trials were likely to give exaggerated effects, with some trials showing an exaggeration of 30 percent.
Rather than continue to rely on statistical significance, the authors wrote, researchers should include these new metrics when reporting future trials.
The study was co-authored by doctoral student Isabelle Weir and Ludovic Trinquart, assistant professor of biostatistics.