Systematic reviewers' perspectives on replication of systematic reviews: A survey

Phi Yen Nguyen, Joanne E. McKenzie, Daniel G. Hamilton, David Moher, Peter Tugwell, Fiona M. Fidler, Neal R. Haddaway, Julian P.T. Higgins, Raju Kanukula, Sathya Karunananthan, Lara Maxwell, Steve McDonald, Shinichi Nakagawa, David Nunan, Vivian Welch, Matthew J. Page

Research output: Contribution to journalArticleResearchpeer-review

Abstract

Background
Replication is essential to the scientific method. It is unclear what systematic reviewers think about the replication of systematic reviews (SRs). Therefore, we aimed to explore systematic reviewers' perspectives on (a) the definition and importance of SR replication; (b) incentives and barriers to conducting SR replication; and (c) a checklist to guide when to replicate an SR.
Methods
We searched PubMed for SRs published from January to April 2021, from which we randomly allocated 50% to this survey and 50% to another survey on data sharing in SRs. We sent an electronic survey to authors of these SRs (n = 4669) using Qualtrics. Quantitative responses were summarized using frequency analysis. Free-text answers were coded using an inductive approach.
Results
The response rate was 9% (n = 409). Most participants considered “replication of SRs” as redoing an SR (68%) or reanalyzing originally collected data (61%), using the same or similar methods. Participants also considered updating an SR, either one's own (42%) or others (43%), equivalent to replication. Most participants agreed that replication of SRs is important (89%). Although 54% of participants reported having conducted a replication of a SR, only 22% have published a replication within 5 years. Those who published a replication (n = 89) often found their replication supported (47%) or expanded the generalizability of the original review (51%). The most common perceived barriers to replicating SRs were difficulty publishing (75%), less prestige (65%), fewer citations (56%), and less impact on career advancement (55%) compared to conducting an original SR. A checklist to assess the need for replication was deemed useful (79%) and easy to apply in practice (69%) by participants.
Conclusion
Reviewers have various perceptions of what constitutes a replication of SRs. Reviewers see replication as important and valuable but perceive several barriers to conducting replications. Institutional support should be better communicated to reviewers to address these perceptions.
Original languageEnglish
Article numbere12009
Number of pages15
JournalCochrane Evidence Synthesis and Methods
Volume1
Issue number2
DOIs
Publication statusPublished - Apr 2023

Cite this