Assessing risk of bias due to missing results in a synthesis

Matthew Page, Jonathan A C Sterne, Julian P T Higgins

Research output: Chapter in Book/Report/Conference proceedingChapter (Book)Otherpeer-review

Abstract

Key Points: Systematic reviews seek to identify all research that meets the eligibility criteria. However, this goal can be compromised by ‘non-reporting bias’: when decisions about how, when or where to report results of eligible studies are influenced by the P value, magnitude or direction of the results. There is convincing evidence for several types of non-reporting bias, reinforcing the need for review authors to search all possible sources where study reports and results may be located. It may be necessary to consult multiple bibliographic databases, trials registers, manufacturers, regulators and study authors or sponsors. Regardless of whether an entire study report or a particular study result is unavailable selectively (e.g. because the P value, magnitude or direction of the results were considered unfavourable by the investigators), the same consequence can arise: risk of bias in a synthesis because available results differ systematically from missing results. Several approaches for assessing risk of bias due to missing results have been suggested. A thorough assessment of selective non-reporting or under-reporting of results in the studies identified is likely to be the most valuable. Because the number of identified studies that have results missing for a given synthesis is known, the impact of selective non-reporting or under-reporting of results can be quantified more easily than the impact of selective non-publication of an unknown number of studies. Funnel plots (and the tests used for examining funnel plot asymmetry) may help review authors to identify evidence of non-reporting biases in cases where protocols or trials register records were unavailable for most studies. However, they have well documented limitations. When there is evidence of funnel plot asymmetry, non-reporting biases should be considered as only one of a number of possible explanations. In these circumstances, review authors should attempt to understand the source(s) of the asymmetry, and consider their implications in the light of any qualitative signals that raise a suspicion of additional missing results, and other sensitivity analyses.
Original languageEnglish
Title of host publicationCochrane Handbook for Systematic Reviews of Interventions
EditorsJulian P. T. Higgins, James Thomas, Jacqueline Chandler, Miranda Cumpston, Tianjing Li, Matthew J. Page, Vivian A. Welch
Place of PublicationChichester UK
PublisherWiley-Blackwell
Chapter13
Pages349-374
Number of pages26
Edition2nd
ISBN (Electronic)9781119536659
ISBN (Print)9781119536628, 9781119536611
Publication statusPublished - Sep 2019

Cite this

Page, M., Sterne, J. A. C., & Higgins, J. P. T. (2019). Assessing risk of bias due to missing results in a synthesis. In J. P. T. Higgins, J. Thomas, J. Chandler, M. Cumpston, T. Li, M. J. Page, & V. A. Welch (Eds.), Cochrane Handbook for Systematic Reviews of Interventions (2nd ed., pp. 349-374). Wiley-Blackwell.