TY - JOUR
T1 - Synthesis methods other than meta-analysis were commonly used but seldom specified
T2 - survey of systematic reviews
AU - Cumpston, Miranda S.
AU - Brennan, Sue E.
AU - Ryan, Rebecca
AU - McKenzie, Joanne E.
N1 - Funding Information:
We acknowledge James Thomas at the EPPI-Centre, UCL Social Research Institute, University College London, for his input into the planning stages of this project, and the assistance of Kristin Read, Research Coordinator at Health Evidence, McMaster University, in providing access to search results from that database. MSC receives funding from the Australian Government Research Training Program. JEM is supported by a National Health and Medical Research Council Investigator Grant (GNT2009612). SEB's position at Cochrane Australia is funded by the Australian Government through the National Health and Medical Research Council. RR's position at Cochrane Consumers and Communication is supported by funding from the Australian Government through the National Health and Medical Research Council. Funding organisations had no role in the conduct or reporting of this study. The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Funding Information:
We acknowledge James Thomas at the EPPI-Centre, UCL Social Research Institute, University College London, for his input into the planning stages of this project, and the assistance of Kristin Read, Research Coordinator at Health Evidence, McMaster University, in providing access to search results from that database. MSC receives funding from the Australian Government Research Training Program. JEM is supported by a National Health and Medical Research Council Investigator Grant ( GNT2009612 ). SEB's position at Cochrane Australia is funded by the Australian Government through the National Health and Medical Research Council. RR's position at Cochrane Consumers and Communication is supported by funding from the Australian Government through the National Health and Medical Research Council. Funding organisations had no role in the conduct or reporting of this study.
Publisher Copyright:
© 2023 The Authors
PY - 2023/4
Y1 - 2023/4
N2 - Objectives: To examine the specification and use of summary and statistical synthesis methods, focusing on synthesis methods other than meta-analysis. Study Design and Setting: We coded the specification and use of summary and synthesis methods in 100 randomly sampled systematic reviews (SRs) of public health and health systems interventions published in 2018 from the Health Evidence and Health Systems Evidence databases. Results: Sixty of the 100 SRs used other synthesis methods for some (27/100) or all syntheses (33/100). Of these, 54/60 used vote counting: three based on direction of effect, 36 on statistical significance, and 15 were unclear. Eight SRs summarized effect estimates (for example, using medians). Seventeen SRs used the term ‘narrative synthesis’ (or equivalent) without describing methods; in practice 15 of these used vote counting. 58/100 SRs used meta-analysis. In SRs providing a rationale for not proceeding with meta-analysis, the most common reason was due to diversity in study characteristics (33/39). Conclusion: Statistical synthesis methods other than meta-analysis are commonly used, but few SRs describe the methods. Improved description of methods is required to allow users to appropriately interpret findings, critique methods used and verify the results. Greater awareness of the serious limitations of vote counting based on statistical significance is required.
AB - Objectives: To examine the specification and use of summary and statistical synthesis methods, focusing on synthesis methods other than meta-analysis. Study Design and Setting: We coded the specification and use of summary and synthesis methods in 100 randomly sampled systematic reviews (SRs) of public health and health systems interventions published in 2018 from the Health Evidence and Health Systems Evidence databases. Results: Sixty of the 100 SRs used other synthesis methods for some (27/100) or all syntheses (33/100). Of these, 54/60 used vote counting: three based on direction of effect, 36 on statistical significance, and 15 were unclear. Eight SRs summarized effect estimates (for example, using medians). Seventeen SRs used the term ‘narrative synthesis’ (or equivalent) without describing methods; in practice 15 of these used vote counting. 58/100 SRs used meta-analysis. In SRs providing a rationale for not proceeding with meta-analysis, the most common reason was due to diversity in study characteristics (33/39). Conclusion: Statistical synthesis methods other than meta-analysis are commonly used, but few SRs describe the methods. Improved description of methods is required to allow users to appropriately interpret findings, critique methods used and verify the results. Greater awareness of the serious limitations of vote counting based on statistical significance is required.
KW - Evidence synthesis
KW - Meta-analysis
KW - Meta-research
KW - Narrative synthesis
KW - Synthesis without meta-analysis
KW - Systematic review
UR - http://www.scopus.com/inward/record.url?scp=85150867277&partnerID=8YFLogxK
U2 - 10.1016/j.jclinepi.2023.02.003
DO - 10.1016/j.jclinepi.2023.02.003
M3 - Article
C2 - 36758885
AN - SCOPUS:85150867277
SN - 0895-4356
VL - 156
SP - 42
EP - 52
JO - Journal of Clinical Epidemiology
JF - Journal of Clinical Epidemiology
ER -
