Differences in Meta-Analyses’ Conclusions Depend on Methods.
Meta-analysis—a research method that aggregates multiple studies on a given topic—is a powerful tool in public health. But the way such analyses are conducted can influence their conclusions, a team of School of Public Health researchers has found.
In an article published in PLOS ONE, researchers led by Christopher Gill, associate professor of global health, found that meta-analyses conducted via the Cochrane Collaboration, which follow a standardized set of methods, produced different results than non-Cochrane meta-analyses on the same topics. In addition, the overlap among studies used in the two kinds of reviews was “surprisingly low,” raising questions about whether the two methods had different criteria for inclusion.
“Our results indicate a substantial divide between the Cochrane and non-Cochrane literature,” Gill said. “While our analysis covers a very small fraction of the vast body of work that comprises the meta-analytic literature, readers should be aware that the two types of meta-analyses are not synonymous—and that in some cases, the discrepancies could lead to fundamentally different conclusions about whether a given intervention is effective or not.”
The study found that non-Cochrane reviews reported significantly higher effect sizes of interventions, with lower precision—a systemic difference that indicates that such reviews “may be overstating the evidence,” Gill said. Meta-analyses conducted via the Cochrane Collaboration adhere to strict methodological and reporting standards that aim to minimize bias, maximize transparency, and improve the accuracy of summarized data, the authors said.
Gill and colleagues conducted a “matched-pair” analysis of 40 meta-analyses related to cardiovascular health, identifying pairs that addressed the same interventions and outcomes. The two sets were similar in terms of publication dates, how many studies were included, and average sample size. But of the 344 studies included in the paired reviews, only 129 (37.5 percent) were included in both kinds of analyses—and only 2 of the 40 matched pairs included exactly the same studies.
Overall, 37.5 percent of the pairs had discrepant results—a finding that the authors called troubling.
“A number of reviews reported markedly different effect sizes, or reported effect sizes that contradicted each other (as from a protective effect to a null or harmful one, or vice versa). These are all instances where the bottom line interpretation by a reader of a review could differ qualitatively,” they wrote.
Gill said that, because meta-analyses are important tools in clinical research, there is a need for such analyses conducted outside of the rigorous Cochrane Collaboration. Still, “it is concerning when two meta-analyses addressing the same question, within a similar time frame, reach different conclusions,” he said. “How should the average doctor or health policy maker react when two ‘gold standards’ disagree with each other? That is the question we have to think about.”
Co-authors on the study included Michael LaValley, professor of biostatistics; doctoral students Alana Brennan and Omid Ameli; and recent MPH graduates Johanna Useem, Michelle Vickery, and Nichole Reinen.