Contradicted and initially stronger effects in highly cited clinical research.

John P A Ioannidis.

Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece. jioannid@cc.uoi.gr

JAMA. 2005 Jul 13;294(2):218-28.

To explore the future contradiction of highly-cited research Ioannidis investigated just under 50 of the most significant and highly regarded medical research findings from 1990 to 2003. Of 45 that concluded their interventions were effective, 34 had had their hypothesis retested. Of these 34, over 40% (14) were subsequently shown to be incorrect or exaggerated. Forty percent of some of the most highly regarded, practice-changing medical evidence from the 20th century subsequently disproven!

summary
ContextControversy and uncertainty ensue when the results of clinical research on the effectiveness of interventions are subsequently contradicted. Controversies are most prominent when high-impact research is involved.ObjectivesTo understand how frequently highly cited studies are contradicted or find effects that are stronger than in other similar studies and to discern whether specific characteristics are associated with such refutation over time.DesignAll original clinical research studies published in 3 major general clinical journals or high-impact-factor specialty journals in 1990-2003 and cited more than 1000 times in the literature were examined.Main Outcome MeasureThe results of highly cited articles were compared against subsequent studies of comparable or larger sample size and similar or better controlled designs. The same analysis was also performed comparatively for matched studies that were not so highly cited.ResultsOf 49 highly cited original clinical research studies, 45 claimed that the intervention was effective. Of these, 7 (16%) were contradicted by subsequent studies, 7 others (16%) had found effects that were stronger than those of subsequent studies, 20 (44%) were replicated, and 11 (24%) remained largely unchallenged. Five of 6 highly-cited nonrandomized studies had been contradicted or had found stronger effects vs 9 of 39 randomized controlled trials (P = .008). Among randomized trials, studies with contradicted or stronger effects were smaller (P = .009) than replicated or unchallenged studies although there was no statistically significant difference in their early or overall citation impact. Matched control studies did not have a significantly different share of refuted results than highly cited studies, but they included more studies with "negative" results.ConclusionsContradiction and initially stronger effects are not unusual in highly cited research of clinical interventions and their outcomes. The extent to which high citations may provoke contradictions and vice versa needs more study. Controversies are most common with highly cited nonrandomized studies, but even the most highly cited randomized trials may be challenged and refuted over time, especially small ones.

  Pubmed   |     Full text   |     Add institutional full text...

Notes

summary
1

To explore the future contradiction of highly-cited research Ioannidis investigated just under 50 of the most significant and highly regarded medical research findings from 1990 to 2003. Of 45 that concluded their interventions were effective, 34 had had their hypothesis retested. Of these 34, over 40% (14) were subsequently shown to be incorrect or exaggerated. Forty percent of some of the most highly regarded, practice-changing medical evidence from the 20th century subsequently disproven!

Daniel Jolley  Daniel Jolley

What will the 'Medical Journal of You' look like?

Start your free 28 day trial now.

We guarantee your privacy. Your email address will not be shared.