Discrepancies in meta-analyses answering the same clinical question were hard to explain: a meta-epidemiological study

Claudia Hacke; David Nunan

doi:10.1016/j.jclinepi.2019.11.015

Discrepancies in meta-analyses answering the same clinical question were hard to explain: a meta-epidemiological study

J Clin Epidemiol. 2020 Mar:119:47-56. doi: 10.1016/j.jclinepi.2019.11.015. Epub 2019 Nov 26.

Authors

Claudia Hacke¹, David Nunan²

Affiliations

¹ Department of Pediatrics I, University Medical Center Schleswig-Holstein, Campus Kiel, Arnold-Heller-Straße 3, Kiel 24105, Germany; Christian-Albrechts-University of Kiel, Institute of Sports Science, Olshausenstraße 74, 24118 Kiel, Germany. Electronic address: claudia.hacke@uksh.de.
² Nuffield Department of Primary Care Health Sciences, University of Oxford, Centre for Evidence-Based Medicine, Radcliffe Observatory Quarter, Woodstock Road, Oxford, OX2 6GG UK.

PMID: 31783099
DOI: 10.1016/j.jclinepi.2019.11.015

Abstract

Objectives: To systematically explore the methodological factors underpinning discrepancies in the pooled effect estimates from Cochrane reviews (CRs) and non-Cochrane reviews (NCRs) systematic reviews, answering the same clinical question.

Study design and setting: Quantitative and qualitative analysis of concordance in effect estimates between meta-analyses from CR and NCR matched on population, intervention, condition, and outcome.

Results: We identified 24 matched meta-analyses from 24 CR to 20 NCR reviews (545 randomized controlled trials [RCTs]). Compared to their CR matched-pair, pooled effects from NCR were the same in only one pair, were on average 0.12 log units (13%) higher (P = 0.012), and had a greater than twofold larger effect size in four matched-pairs. Two-thirds of CR (15/24, 70.8%) and 0/20 (0%) NCR were rated to have moderate to high confidence in their results (AMSTAR 2). Differences in pre-defined methods, including search strategy, eligibility criteria, and performance of dual screening, could explain mismatches in included studies. Disagreements in the interpretation of eligibility criteria were identified as reasons underpinning discrepant findings in 14 pairs. 23/24 meta-analyses included at least one study of its match. Only two pairs agreed on the numerical data presented for the same studies. An assessment of 50% of discrepant studies (n = 45) showed that reasons for differences in extracted data could be identified in 15 studies.

Conclusion: On average, meta-analyses from NCR reported higher effect estimates compared with meta-analyses from CR answering the same clinical question. Methodological and author judgments and performance are key aspects underpinning poor overlap of included studies and discrepancies in reported effect estimates. The potential impacts on health care policy and clinical practice are far-reaching but still remain unknown. Reinforcing awareness and scrutiny of application of reporting guidelines and improvements in protocol registration are needed.

Keywords: Cochrane review; Discrepancies; Matched-pair analysis; Meta-analysis; Methodological quality; Non-Cochrane review; Systematic review.

MeSH terms

Epidemiologic Research Design*
Epidemiologic Studies*
Humans
Meta-Analysis as Topic*