Nonrandomized studies using causal-modeling may give different answers than RCTs: a meta-epidemiological study

Hannah Ewald; John P A Ioannidis; Aviv Ladanie; Kimberly Mc Cord; Heiner C Bucher; Lars G Hemkens

doi:10.1016/j.jclinepi.2019.10.012

Nonrandomized studies using causal-modeling may give different answers than RCTs: a meta-epidemiological study

J Clin Epidemiol. 2020 Feb:118:29-41. doi: 10.1016/j.jclinepi.2019.10.012. Epub 2019 Nov 5.

Authors

Hannah Ewald¹, John P A Ioannidis², Aviv Ladanie³, Kimberly Mc Cord³, Heiner C Bucher⁴, Lars G Hemkens⁵

Affiliations

¹ Department of Clinical Research, Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, University of Basel, 4031 Basel, Switzerland; Swiss Tropical and Public Health Institute, University of Basel, 4051 Basel, Switzerland; University Medical Library, University of Basel, Basel, Switzerland.
² Department of Medicine, Stanford Prevention Research Center, Stanford University School of Medicine, Stanford, CA 94305, USA; Meta-Research Innovation Center at Stanford (METRICS), Stanford School of Medicine, Palo Alto, CA 94304, USA; Department of Health Research and Policy, Stanford University School of Medicine, Stanford, CA 94305, USA; Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA 94305, USA; Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA 94305, USA.
³ Department of Clinical Research, Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, University of Basel, 4031 Basel, Switzerland; Swiss Tropical and Public Health Institute, University of Basel, 4051 Basel, Switzerland.
⁴ Department of Clinical Research, Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, University of Basel, 4031 Basel, Switzerland.
⁵ Department of Clinical Research, Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, University of Basel, 4031 Basel, Switzerland. Electronic address: Lars.Hemkens@usb.ch.

PMID: 31704350
DOI: 10.1016/j.jclinepi.2019.10.012

Abstract

Objectives: To evaluate how estimated treatment effects agree between nonrandomized studies using causal modeling with marginal structural models (MSM-studies) and randomized trials (RCTs).

Study design: Meta-epidemiological study.

Setting: MSM-studies providing effect estimates on any healthcare outcome of any treatment were eligible. We systematically sought RCTs on the same clinical question and compared the direction of treatment effects, effect sizes, and confidence intervals.

Results: The main analysis included 19 MSM-studies (1,039,570 patients) and 141 RCTs (120,669 patients). MSM-studies indicated effect estimates in the opposite direction from RCTs for eight clinical questions (42%), and their 95% CI (confidence interval) did not include the RCT estimate in nine clinical questions (47%). The effect estimates deviated 1.58-fold between the study designs (median absolute deviation OR [odds ratio] 1.58; IQR [interquartile range] 1.37 to 2.16). Overall, we found no systematic disagreement regarding benefit or harm but confidence intervals were wide (summary ratio of odds ratios [sROR] 1.04; 95% CI 0.88 to 1.23). The subset of MSM-studies focusing on healthcare decision-making tended to overestimate experimental treatment benefits (sROR 1.44; 95% CI 0.99 to 2.09).

Conclusion: Nonrandomized studies using causal modeling with MSM may give different answers than RCTs. Caution is still required when nonrandomized "real world" evidence is used for healthcare decisions.

Keywords: Clinical decision-making; Confounding; Meta-analysis; Methodology; Statistical models; Systematic review.

Publication types

Research Support, Non-U.S. Gov't
Review

MeSH terms

Clinical Decision-Making
Data Interpretation, Statistical*
Epidemiologic Studies*
Humans
Meta-Analysis as Topic*
Models, Statistical*
Outcome Assessment, Health Care
Randomized Controlled Trials as Topic