Flaws in the application and interpretation of statistical analyses in systematic reviews of therapeutic interventions were common

a cross-sectional analysis

Matthew J. Page, Douglas G. Altman, Joanne E. McKenzie, Larissa Shamseer, Nadera Ahmadzai, Dianna Wolfe, Fatemeh Yazdi, Ferrán Catalá-López, Andrea C. Tricco, David Moher

Research output: Contribution to journalArticleResearchpeer-review

6 Citations (Scopus)

Abstract

Objectives: The objective of the study was to investigate the application and interpretation of statistical analyses in a cross-section of systematic reviews (SRs) of therapeutic interventions, without restriction by journal, clinical condition, or specialty. Study Design and Setting: We evaluated a random sample of SRs assembled previously, which were indexed in MEDLINE® during February 2014, focused on a treatment or prevention question, and reported at least one meta-analysis. The reported statistical methods used in each SR were extracted from articles and online appendices by one author, with a 20% random sample extracted in duplicate. Results: We evaluated 110 SRs; 78/110 (71%) were non-Cochrane SRs and 55/110 (50%) investigated a pharmacological intervention. The SRs presented a median of 13 (interquartile range: 5–27) meta-analytic effects. When considering the index (primary or first reported) meta-analysis of each SR, just over half (62/110 [56%]) used the random-effects model, but few (5/62 [8%]) interpreted the meta-analytic effect correctly (as the average of the intervention effects across all studies). A statistical test for funnel plot asymmetry was reported in 17/110 (15%) SRs; however, in only 4/17 (24%) did the test include the recommended number of at least 10 studies of varying size. Subgroup analyses accompanied 42/110 (38%) index meta-analyses, but findings were not interpreted with respect to a test for interaction in 29/42 (69%) cases, and the issue of potential confounding in the subgroup analyses was not raised in any SR. Conclusions: There is scope for improvement in the application and interpretation of statistical analyses in SRs of therapeutic interventions. The involvement of statisticians on the SR team and establishment of partnerships between researchers with specialist expertise in SR methods and journal editors may help overcome these shortcomings.

Original languageEnglish
Pages (from-to)7-18
Number of pages12
JournalJournal of Clinical Epidemiology
Volume95
DOIs
Publication statusPublished - 1 Mar 2018

Keywords

  • Meta-analysis
  • Meta-research
  • Methodology
  • Quality
  • Reporting
  • Systematic reviews

Cite this

Page, Matthew J. ; Altman, Douglas G. ; McKenzie, Joanne E. ; Shamseer, Larissa ; Ahmadzai, Nadera ; Wolfe, Dianna ; Yazdi, Fatemeh ; Catalá-López, Ferrán ; Tricco, Andrea C. ; Moher, David. / Flaws in the application and interpretation of statistical analyses in systematic reviews of therapeutic interventions were common : a cross-sectional analysis. In: Journal of Clinical Epidemiology. 2018 ; Vol. 95. pp. 7-18.
@article{e7de5f68f2b5495b9a9ba432d26422fc,
title = "Flaws in the application and interpretation of statistical analyses in systematic reviews of therapeutic interventions were common: a cross-sectional analysis",
abstract = "Objectives: The objective of the study was to investigate the application and interpretation of statistical analyses in a cross-section of systematic reviews (SRs) of therapeutic interventions, without restriction by journal, clinical condition, or specialty. Study Design and Setting: We evaluated a random sample of SRs assembled previously, which were indexed in MEDLINE{\circledR} during February 2014, focused on a treatment or prevention question, and reported at least one meta-analysis. The reported statistical methods used in each SR were extracted from articles and online appendices by one author, with a 20{\%} random sample extracted in duplicate. Results: We evaluated 110 SRs; 78/110 (71{\%}) were non-Cochrane SRs and 55/110 (50{\%}) investigated a pharmacological intervention. The SRs presented a median of 13 (interquartile range: 5–27) meta-analytic effects. When considering the index (primary or first reported) meta-analysis of each SR, just over half (62/110 [56{\%}]) used the random-effects model, but few (5/62 [8{\%}]) interpreted the meta-analytic effect correctly (as the average of the intervention effects across all studies). A statistical test for funnel plot asymmetry was reported in 17/110 (15{\%}) SRs; however, in only 4/17 (24{\%}) did the test include the recommended number of at least 10 studies of varying size. Subgroup analyses accompanied 42/110 (38{\%}) index meta-analyses, but findings were not interpreted with respect to a test for interaction in 29/42 (69{\%}) cases, and the issue of potential confounding in the subgroup analyses was not raised in any SR. Conclusions: There is scope for improvement in the application and interpretation of statistical analyses in SRs of therapeutic interventions. The involvement of statisticians on the SR team and establishment of partnerships between researchers with specialist expertise in SR methods and journal editors may help overcome these shortcomings.",
keywords = "Meta-analysis, Meta-research, Methodology, Quality, Reporting, Systematic reviews",
author = "Page, {Matthew J.} and Altman, {Douglas G.} and McKenzie, {Joanne E.} and Larissa Shamseer and Nadera Ahmadzai and Dianna Wolfe and Fatemeh Yazdi and Ferr{\'a}n Catal{\'a}-L{\'o}pez and Tricco, {Andrea C.} and David Moher",
year = "2018",
month = "3",
day = "1",
doi = "10.1016/j.jclinepi.2017.11.022",
language = "English",
volume = "95",
pages = "7--18",
journal = "Journal of Clinical Epidemiology",
issn = "0895-4356",
publisher = "Elsevier",

}

Flaws in the application and interpretation of statistical analyses in systematic reviews of therapeutic interventions were common : a cross-sectional analysis. / Page, Matthew J.; Altman, Douglas G.; McKenzie, Joanne E.; Shamseer, Larissa; Ahmadzai, Nadera; Wolfe, Dianna; Yazdi, Fatemeh; Catalá-López, Ferrán; Tricco, Andrea C.; Moher, David.

In: Journal of Clinical Epidemiology, Vol. 95, 01.03.2018, p. 7-18.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Flaws in the application and interpretation of statistical analyses in systematic reviews of therapeutic interventions were common

T2 - a cross-sectional analysis

AU - Page, Matthew J.

AU - Altman, Douglas G.

AU - McKenzie, Joanne E.

AU - Shamseer, Larissa

AU - Ahmadzai, Nadera

AU - Wolfe, Dianna

AU - Yazdi, Fatemeh

AU - Catalá-López, Ferrán

AU - Tricco, Andrea C.

AU - Moher, David

PY - 2018/3/1

Y1 - 2018/3/1

N2 - Objectives: The objective of the study was to investigate the application and interpretation of statistical analyses in a cross-section of systematic reviews (SRs) of therapeutic interventions, without restriction by journal, clinical condition, or specialty. Study Design and Setting: We evaluated a random sample of SRs assembled previously, which were indexed in MEDLINE® during February 2014, focused on a treatment or prevention question, and reported at least one meta-analysis. The reported statistical methods used in each SR were extracted from articles and online appendices by one author, with a 20% random sample extracted in duplicate. Results: We evaluated 110 SRs; 78/110 (71%) were non-Cochrane SRs and 55/110 (50%) investigated a pharmacological intervention. The SRs presented a median of 13 (interquartile range: 5–27) meta-analytic effects. When considering the index (primary or first reported) meta-analysis of each SR, just over half (62/110 [56%]) used the random-effects model, but few (5/62 [8%]) interpreted the meta-analytic effect correctly (as the average of the intervention effects across all studies). A statistical test for funnel plot asymmetry was reported in 17/110 (15%) SRs; however, in only 4/17 (24%) did the test include the recommended number of at least 10 studies of varying size. Subgroup analyses accompanied 42/110 (38%) index meta-analyses, but findings were not interpreted with respect to a test for interaction in 29/42 (69%) cases, and the issue of potential confounding in the subgroup analyses was not raised in any SR. Conclusions: There is scope for improvement in the application and interpretation of statistical analyses in SRs of therapeutic interventions. The involvement of statisticians on the SR team and establishment of partnerships between researchers with specialist expertise in SR methods and journal editors may help overcome these shortcomings.

AB - Objectives: The objective of the study was to investigate the application and interpretation of statistical analyses in a cross-section of systematic reviews (SRs) of therapeutic interventions, without restriction by journal, clinical condition, or specialty. Study Design and Setting: We evaluated a random sample of SRs assembled previously, which were indexed in MEDLINE® during February 2014, focused on a treatment or prevention question, and reported at least one meta-analysis. The reported statistical methods used in each SR were extracted from articles and online appendices by one author, with a 20% random sample extracted in duplicate. Results: We evaluated 110 SRs; 78/110 (71%) were non-Cochrane SRs and 55/110 (50%) investigated a pharmacological intervention. The SRs presented a median of 13 (interquartile range: 5–27) meta-analytic effects. When considering the index (primary or first reported) meta-analysis of each SR, just over half (62/110 [56%]) used the random-effects model, but few (5/62 [8%]) interpreted the meta-analytic effect correctly (as the average of the intervention effects across all studies). A statistical test for funnel plot asymmetry was reported in 17/110 (15%) SRs; however, in only 4/17 (24%) did the test include the recommended number of at least 10 studies of varying size. Subgroup analyses accompanied 42/110 (38%) index meta-analyses, but findings were not interpreted with respect to a test for interaction in 29/42 (69%) cases, and the issue of potential confounding in the subgroup analyses was not raised in any SR. Conclusions: There is scope for improvement in the application and interpretation of statistical analyses in SRs of therapeutic interventions. The involvement of statisticians on the SR team and establishment of partnerships between researchers with specialist expertise in SR methods and journal editors may help overcome these shortcomings.

KW - Meta-analysis

KW - Meta-research

KW - Methodology

KW - Quality

KW - Reporting

KW - Systematic reviews

UR - http://www.scopus.com/inward/record.url?scp=85042514658&partnerID=8YFLogxK

U2 - 10.1016/j.jclinepi.2017.11.022

DO - 10.1016/j.jclinepi.2017.11.022

M3 - Article

VL - 95

SP - 7

EP - 18

JO - Journal of Clinical Epidemiology

JF - Journal of Clinical Epidemiology

SN - 0895-4356

ER -