Data and code availability statements in systematic reviews of interventions were often missing or inaccurate: a content analysis

Matthew J. Page; Phi Yen Nguyen; Daniel G. Hamilton; Neal R. Haddaway; Raju Kanukula; David Moher; Joanne E. McKenzie

doi:10.1016/j.jclinepi.2022.03.003

Data and code availability statements in systematic reviews of interventions were often missing or inaccurate: a content analysis

Matthew J. Page
, Phi Yen Nguyen
, Daniel G. Hamilton
, Neal R. Haddaway
, Raju Kanukula
, David Moher
, Joanne E. McKenzie

Research output: Contribution to journal › Article › Research › peer-review

37 Citations (Scopus)

Abstract

Objectives: To estimate the frequency of data and code availability statements in a random sample of systematic reviews with meta-analysis of aggregate data, summarize the content of the statements and investigate how often data and code files were shared. Methods: We searched for systematic reviews with meta-analysis of aggregate data on the effects of a health, social, behavioral, or educational intervention that were indexed in PubMed, Education Collection via ProQuest, Scopus via Elsevier, or Social Sciences Citation Index and Science Citation Index Expanded via Web of Science during a 4-week period (between November 2, and December 2, 2020). Records were randomly sorted and screened independently by two authors until our target sample of 300 systematic reviews was reached. Two authors independently recorded whether a data or code availability statement (or both) appeared in each review and coded the content of the statements using an inductive approach. Results: Of the 300 included systematic reviews with meta-analysis, 86 (29%) had a data availability statement, and seven (2%) had both a data and code availability statement. In 12/93 (13%) data availability statements, authors stated that data files were available for download from the journal website or a data repository, which we verified as being true. While 39/93 (42%) authors stated data were available upon request, 37/93 (40%) implied that sharing of data files was not necessary or applicable to them, most often because “all data appear in the article” or “no datasets were generated or analyzed”. Discussion: Data and code availability statements appear infrequently in systematic review manuscripts. Authors who do provide a data availability statement often incorrectly imply that data sharing is not applicable to systematic reviews. Our results suggest the need for various interventions to increase data and code sharing by systematic reviewers.

Original language	English
Pages (from-to)	1-10
Number of pages	10
Journal	Journal of Clinical Epidemiology
Volume	147
DOIs	https://doi.org/10.1016/j.jclinepi.2022.03.003
Publication status	Published - Jul 2022

Keywords

Evidence synthesis
Open data
Open science
Open synthesis
Reproducibility of research
Research integrity

Access to Document

10.1016/j.jclinepi.2022.03.003

Cite this

@article{1957c243918240bc895e1ce9d09f9eb2,

title = "Data and code availability statements in systematic reviews of interventions were often missing or inaccurate: a content analysis",

abstract = "Objectives: To estimate the frequency of data and code availability statements in a random sample of systematic reviews with meta-analysis of aggregate data, summarize the content of the statements and investigate how often data and code files were shared. Methods: We searched for systematic reviews with meta-analysis of aggregate data on the effects of a health, social, behavioral, or educational intervention that were indexed in PubMed, Education Collection via ProQuest, Scopus via Elsevier, or Social Sciences Citation Index and Science Citation Index Expanded via Web of Science during a 4-week period (between November 2, and December 2, 2020). Records were randomly sorted and screened independently by two authors until our target sample of 300 systematic reviews was reached. Two authors independently recorded whether a data or code availability statement (or both) appeared in each review and coded the content of the statements using an inductive approach. Results: Of the 300 included systematic reviews with meta-analysis, 86 (29\%) had a data availability statement, and seven (2\%) had both a data and code availability statement. In 12/93 (13\%) data availability statements, authors stated that data files were available for download from the journal website or a data repository, which we verified as being true. While 39/93 (42\%) authors stated data were available upon request, 37/93 (40\%) implied that sharing of data files was not necessary or applicable to them, most often because “all data appear in the article” or “no datasets were generated or analyzed”. Discussion: Data and code availability statements appear infrequently in systematic review manuscripts. Authors who do provide a data availability statement often incorrectly imply that data sharing is not applicable to systematic reviews. Our results suggest the need for various interventions to increase data and code sharing by systematic reviewers.",

keywords = "Evidence synthesis, Open data, Open science, Open synthesis, Reproducibility of research, Research integrity",

author = "Page, \{Matthew J.\} and Nguyen, \{Phi Yen\} and Hamilton, \{Daniel G.\} and Haddaway, \{Neal R.\} and Raju Kanukula and David Moher and McKenzie, \{Joanne E.\}",

note = "Funding Information: Funding: This research was funded by an Australian Research Council Discovery Early Career Researcher Award ( DE200101618 ), held by MJP. DGH is supported by an Australian Commonwealth Government Research Training Program Scholarship . NRH is funded by an Alexander von Humboldt Experienced Researcher Fellowship. RK is supported by a Monash Graduate Scholarship and a Monash International Tuition Scholarship. DM is supported in part by a University Research Chair, University of Ottawa. JEM is supported by a National Health and Medical Research Council Career Development Fellowship ( APP1143429 ). The funders had no role in the study design, decision to publish, or preparation of the manuscript. Publisher Copyright: {\textcopyright} 2022 Elsevier Inc.",

year = "2022",

month = jul,

doi = "10.1016/j.jclinepi.2022.03.003",

language = "English",

volume = "147",

pages = "1--10",

journal = "Journal of Clinical Epidemiology",

issn = "0895-4356",

publisher = "Elsevier",

}

TY - JOUR

T1 - Data and code availability statements in systematic reviews of interventions were often missing or inaccurate

T2 - a content analysis

AU - Page, Matthew J.

AU - Nguyen, Phi Yen

AU - Hamilton, Daniel G.

AU - Haddaway, Neal R.

AU - Kanukula, Raju

AU - Moher, David

AU - McKenzie, Joanne E.

N1 - Funding Information: Funding: This research was funded by an Australian Research Council Discovery Early Career Researcher Award ( DE200101618 ), held by MJP. DGH is supported by an Australian Commonwealth Government Research Training Program Scholarship . NRH is funded by an Alexander von Humboldt Experienced Researcher Fellowship. RK is supported by a Monash Graduate Scholarship and a Monash International Tuition Scholarship. DM is supported in part by a University Research Chair, University of Ottawa. JEM is supported by a National Health and Medical Research Council Career Development Fellowship ( APP1143429 ). The funders had no role in the study design, decision to publish, or preparation of the manuscript. Publisher Copyright: © 2022 Elsevier Inc.

PY - 2022/7

Y1 - 2022/7

N2 - Objectives: To estimate the frequency of data and code availability statements in a random sample of systematic reviews with meta-analysis of aggregate data, summarize the content of the statements and investigate how often data and code files were shared. Methods: We searched for systematic reviews with meta-analysis of aggregate data on the effects of a health, social, behavioral, or educational intervention that were indexed in PubMed, Education Collection via ProQuest, Scopus via Elsevier, or Social Sciences Citation Index and Science Citation Index Expanded via Web of Science during a 4-week period (between November 2, and December 2, 2020). Records were randomly sorted and screened independently by two authors until our target sample of 300 systematic reviews was reached. Two authors independently recorded whether a data or code availability statement (or both) appeared in each review and coded the content of the statements using an inductive approach. Results: Of the 300 included systematic reviews with meta-analysis, 86 (29%) had a data availability statement, and seven (2%) had both a data and code availability statement. In 12/93 (13%) data availability statements, authors stated that data files were available for download from the journal website or a data repository, which we verified as being true. While 39/93 (42%) authors stated data were available upon request, 37/93 (40%) implied that sharing of data files was not necessary or applicable to them, most often because “all data appear in the article” or “no datasets were generated or analyzed”. Discussion: Data and code availability statements appear infrequently in systematic review manuscripts. Authors who do provide a data availability statement often incorrectly imply that data sharing is not applicable to systematic reviews. Our results suggest the need for various interventions to increase data and code sharing by systematic reviewers.

AB - Objectives: To estimate the frequency of data and code availability statements in a random sample of systematic reviews with meta-analysis of aggregate data, summarize the content of the statements and investigate how often data and code files were shared. Methods: We searched for systematic reviews with meta-analysis of aggregate data on the effects of a health, social, behavioral, or educational intervention that were indexed in PubMed, Education Collection via ProQuest, Scopus via Elsevier, or Social Sciences Citation Index and Science Citation Index Expanded via Web of Science during a 4-week period (between November 2, and December 2, 2020). Records were randomly sorted and screened independently by two authors until our target sample of 300 systematic reviews was reached. Two authors independently recorded whether a data or code availability statement (or both) appeared in each review and coded the content of the statements using an inductive approach. Results: Of the 300 included systematic reviews with meta-analysis, 86 (29%) had a data availability statement, and seven (2%) had both a data and code availability statement. In 12/93 (13%) data availability statements, authors stated that data files were available for download from the journal website or a data repository, which we verified as being true. While 39/93 (42%) authors stated data were available upon request, 37/93 (40%) implied that sharing of data files was not necessary or applicable to them, most often because “all data appear in the article” or “no datasets were generated or analyzed”. Discussion: Data and code availability statements appear infrequently in systematic review manuscripts. Authors who do provide a data availability statement often incorrectly imply that data sharing is not applicable to systematic reviews. Our results suggest the need for various interventions to increase data and code sharing by systematic reviewers.

KW - Evidence synthesis

KW - Open data

KW - Open science

KW - Open synthesis

KW - Reproducibility of research

KW - Research integrity

UR - https://www.scopus.com/pages/publications/85127327329

U2 - 10.1016/j.jclinepi.2022.03.003

DO - 10.1016/j.jclinepi.2022.03.003

M3 - Article

C2 - 35278609

AN - SCOPUS:85127327329

SN - 0895-4356

VL - 147

SP - 1

EP - 10

JO - Journal of Clinical Epidemiology

JF - Journal of Clinical Epidemiology

ER -

Data and code availability statements in systematic reviews of interventions were often missing or inaccurate: a content analysis

Abstract

Keywords

Access to Document

Other files and links

Reproducibility and transparency in the synthesis of research findings

Methodological research in meta-analysis and evidence synthesis: An evidence-based methods approach

Cite this

Data and code availability statements in systematic reviews of interventions were often missing or inaccurate: a content analysis

Abstract

Keywords

Access to Document

Other files and links

Projects

Reproducibility and transparency in the synthesis of research findings

Methodological research in meta-analysis and evidence synthesis: An evidence-based methods approach

Cite this