Reproducible research practices are underused in systematic reviews of biomedical interventions

Matthew J. Page, Douglas G. Altman, Larissa Shamseer, Joanne E. McKenzie, Nadera Ahmadzai, Dianna Wolfe, Fatemeh Yazdi, Ferrán Catalá-López, Andrea C. Tricco, David Moher

Research output: Contribution to journalReview ArticleResearchpeer-review

10 Citations (Scopus)

Abstract

Objectives: To evaluate how often reproducible research practices, which allow others to recreate the findings of studies, given the original data, are used in systematic reviews (SRs) of biomedical research. Study Design and Setting: We evaluated a random sample of SRs indexed in MEDLINE during February 2014, which focused on a therapeutic intervention and reported at least one meta-analysis. Data on reproducible research practices in each SR were extracted using a 26-item form by one author, with a 20% random sample extracted in duplicate. We explored whether the use of reproducible research practices was associated with an SR being a Cochrane review, as well as with the reported use of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. Results: We evaluated 110 SRs of therapeutic interventions, 78 (71%) of which were non-Cochrane SRs. Across the SRs, there were 2,139 meta-analytic effects (including subgroup meta-analytic effects and sensitivity analyses), 1,551 (73%) of which were reported in sufficient detail to recreate them. Systematic reviewers reported the data needed to recreate all meta-analytic effects in 72 (65%) SRs only. This percentage was higher in Cochrane than in non-Cochrane SRs (30/32 [94%] vs. 42/78 [54%]; risk ratio 1.74, 95% confidence interval 1.39–2.18). Systematic reviewers who reported imputing, algebraically manipulating, or obtaining some data from the study author/sponsor infrequently stated which specific data were handled in this way. Only 33 (30%) SRs mentioned access to data sets and statistical code used to perform analyses. Conclusion: Reproducible research practices are underused in SRs of biomedical interventions. Adoption of such practices facilitates identification of errors and allows the SR data to be reanalyzed.

Original languageEnglish
Pages (from-to)8-18
Number of pages11
JournalJournal of Clinical Epidemiology
Volume94
DOIs
Publication statusPublished - 1 Feb 2018

Keywords

  • Data sharing
  • Methodology
  • Quality
  • Reporting
  • Reproducibility
  • Systematic reviews

Cite this

Page, Matthew J. ; Altman, Douglas G. ; Shamseer, Larissa ; McKenzie, Joanne E. ; Ahmadzai, Nadera ; Wolfe, Dianna ; Yazdi, Fatemeh ; Catalá-López, Ferrán ; Tricco, Andrea C. ; Moher, David. / Reproducible research practices are underused in systematic reviews of biomedical interventions. In: Journal of Clinical Epidemiology. 2018 ; Vol. 94. pp. 8-18.
@article{c07e134697104c5d98e32414f63a3b9f,
title = "Reproducible research practices are underused in systematic reviews of biomedical interventions",
abstract = "Objectives: To evaluate how often reproducible research practices, which allow others to recreate the findings of studies, given the original data, are used in systematic reviews (SRs) of biomedical research. Study Design and Setting: We evaluated a random sample of SRs indexed in MEDLINE during February 2014, which focused on a therapeutic intervention and reported at least one meta-analysis. Data on reproducible research practices in each SR were extracted using a 26-item form by one author, with a 20{\%} random sample extracted in duplicate. We explored whether the use of reproducible research practices was associated with an SR being a Cochrane review, as well as with the reported use of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. Results: We evaluated 110 SRs of therapeutic interventions, 78 (71{\%}) of which were non-Cochrane SRs. Across the SRs, there were 2,139 meta-analytic effects (including subgroup meta-analytic effects and sensitivity analyses), 1,551 (73{\%}) of which were reported in sufficient detail to recreate them. Systematic reviewers reported the data needed to recreate all meta-analytic effects in 72 (65{\%}) SRs only. This percentage was higher in Cochrane than in non-Cochrane SRs (30/32 [94{\%}] vs. 42/78 [54{\%}]; risk ratio 1.74, 95{\%} confidence interval 1.39–2.18). Systematic reviewers who reported imputing, algebraically manipulating, or obtaining some data from the study author/sponsor infrequently stated which specific data were handled in this way. Only 33 (30{\%}) SRs mentioned access to data sets and statistical code used to perform analyses. Conclusion: Reproducible research practices are underused in SRs of biomedical interventions. Adoption of such practices facilitates identification of errors and allows the SR data to be reanalyzed.",
keywords = "Data sharing, Methodology, Quality, Reporting, Reproducibility, Systematic reviews",
author = "Page, {Matthew J.} and Altman, {Douglas G.} and Larissa Shamseer and McKenzie, {Joanne E.} and Nadera Ahmadzai and Dianna Wolfe and Fatemeh Yazdi and Ferr{\'a}n Catal{\'a}-L{\'o}pez and Tricco, {Andrea C.} and David Moher",
year = "2018",
month = "2",
day = "1",
doi = "10.1016/j.jclinepi.2017.10.017",
language = "English",
volume = "94",
pages = "8--18",
journal = "Journal of Clinical Epidemiology",
issn = "0895-4356",
publisher = "Elsevier",

}

Page, MJ, Altman, DG, Shamseer, L, McKenzie, JE, Ahmadzai, N, Wolfe, D, Yazdi, F, Catalá-López, F, Tricco, AC & Moher, D 2018, 'Reproducible research practices are underused in systematic reviews of biomedical interventions', Journal of Clinical Epidemiology, vol. 94, pp. 8-18. https://doi.org/10.1016/j.jclinepi.2017.10.017

Reproducible research practices are underused in systematic reviews of biomedical interventions. / Page, Matthew J.; Altman, Douglas G.; Shamseer, Larissa; McKenzie, Joanne E.; Ahmadzai, Nadera; Wolfe, Dianna; Yazdi, Fatemeh; Catalá-López, Ferrán; Tricco, Andrea C.; Moher, David.

In: Journal of Clinical Epidemiology, Vol. 94, 01.02.2018, p. 8-18.

Research output: Contribution to journalReview ArticleResearchpeer-review

TY - JOUR

T1 - Reproducible research practices are underused in systematic reviews of biomedical interventions

AU - Page, Matthew J.

AU - Altman, Douglas G.

AU - Shamseer, Larissa

AU - McKenzie, Joanne E.

AU - Ahmadzai, Nadera

AU - Wolfe, Dianna

AU - Yazdi, Fatemeh

AU - Catalá-López, Ferrán

AU - Tricco, Andrea C.

AU - Moher, David

PY - 2018/2/1

Y1 - 2018/2/1

N2 - Objectives: To evaluate how often reproducible research practices, which allow others to recreate the findings of studies, given the original data, are used in systematic reviews (SRs) of biomedical research. Study Design and Setting: We evaluated a random sample of SRs indexed in MEDLINE during February 2014, which focused on a therapeutic intervention and reported at least one meta-analysis. Data on reproducible research practices in each SR were extracted using a 26-item form by one author, with a 20% random sample extracted in duplicate. We explored whether the use of reproducible research practices was associated with an SR being a Cochrane review, as well as with the reported use of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. Results: We evaluated 110 SRs of therapeutic interventions, 78 (71%) of which were non-Cochrane SRs. Across the SRs, there were 2,139 meta-analytic effects (including subgroup meta-analytic effects and sensitivity analyses), 1,551 (73%) of which were reported in sufficient detail to recreate them. Systematic reviewers reported the data needed to recreate all meta-analytic effects in 72 (65%) SRs only. This percentage was higher in Cochrane than in non-Cochrane SRs (30/32 [94%] vs. 42/78 [54%]; risk ratio 1.74, 95% confidence interval 1.39–2.18). Systematic reviewers who reported imputing, algebraically manipulating, or obtaining some data from the study author/sponsor infrequently stated which specific data were handled in this way. Only 33 (30%) SRs mentioned access to data sets and statistical code used to perform analyses. Conclusion: Reproducible research practices are underused in SRs of biomedical interventions. Adoption of such practices facilitates identification of errors and allows the SR data to be reanalyzed.

AB - Objectives: To evaluate how often reproducible research practices, which allow others to recreate the findings of studies, given the original data, are used in systematic reviews (SRs) of biomedical research. Study Design and Setting: We evaluated a random sample of SRs indexed in MEDLINE during February 2014, which focused on a therapeutic intervention and reported at least one meta-analysis. Data on reproducible research practices in each SR were extracted using a 26-item form by one author, with a 20% random sample extracted in duplicate. We explored whether the use of reproducible research practices was associated with an SR being a Cochrane review, as well as with the reported use of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. Results: We evaluated 110 SRs of therapeutic interventions, 78 (71%) of which were non-Cochrane SRs. Across the SRs, there were 2,139 meta-analytic effects (including subgroup meta-analytic effects and sensitivity analyses), 1,551 (73%) of which were reported in sufficient detail to recreate them. Systematic reviewers reported the data needed to recreate all meta-analytic effects in 72 (65%) SRs only. This percentage was higher in Cochrane than in non-Cochrane SRs (30/32 [94%] vs. 42/78 [54%]; risk ratio 1.74, 95% confidence interval 1.39–2.18). Systematic reviewers who reported imputing, algebraically manipulating, or obtaining some data from the study author/sponsor infrequently stated which specific data were handled in this way. Only 33 (30%) SRs mentioned access to data sets and statistical code used to perform analyses. Conclusion: Reproducible research practices are underused in SRs of biomedical interventions. Adoption of such practices facilitates identification of errors and allows the SR data to be reanalyzed.

KW - Data sharing

KW - Methodology

KW - Quality

KW - Reporting

KW - Reproducibility

KW - Systematic reviews

UR - http://www.scopus.com/inward/record.url?scp=85040199860&partnerID=8YFLogxK

U2 - 10.1016/j.jclinepi.2017.10.017

DO - 10.1016/j.jclinepi.2017.10.017

M3 - Review Article

VL - 94

SP - 8

EP - 18

JO - Journal of Clinical Epidemiology

JF - Journal of Clinical Epidemiology

SN - 0895-4356

ER -