Dealing with effect size multiplicity in systematic reviews and meta-analyses

José A. López-López, Matthew J. Page, Mark W. Lipsey, Julian P.T. Higgins

Research output: Contribution to journalArticleOtherpeer-review

3 Citations (Scopus)

Abstract

Systematic reviews often encounter primary studies that report multiple effect sizes based on data from the same participants. These have the potential to introduce statistical dependency into the meta-analytic data set. In this paper, we provide a tutorial on dealing with effect size multiplicity within studies in the context of meta-analyses of intervention and association studies, recommending a three-step approach. The first step is to define the research question and consider the extent to which it mainly reflects interest in mean effect sizes (which we term a convergent approach) or an interest in exploring heterogeneity (which we term a divergent approach). A second step is to identify the types of multiplicities that appear in the initial database of effect sizes relevant to the research question, and we propose a categorization scheme to differentiate them. The third step is to select a strategy for dealing with each type of multiplicity. The researcher can choose between a reductionist meta-analytic approach, which is characterized by inclusion of a single effect size per study, and an integrative approach, characterized by inclusion of multiple effect sizes per study. We present an overview of available analysis strategies for dealing with effect size multiplicity within studies and provide recommendations intended to help researchers decide which strategy might be preferable in particular situations. Last, we offer caveats and cautions about addressing the challenges multiplicity poses for systematic reviews and meta-analyses.

Original languageEnglish
Pages (from-to)336-351
Number of pages16
JournalResearch Synthesis Methods
Volume9
Issue number3
DOIs
Publication statusPublished - Sep 2018

Keywords

  • Dependency
  • Effect size
  • Meta-analysis
  • Multiplicity
  • Systematic review

Cite this

López-López, José A. ; Page, Matthew J. ; Lipsey, Mark W. ; Higgins, Julian P.T. / Dealing with effect size multiplicity in systematic reviews and meta-analyses. In: Research Synthesis Methods. 2018 ; Vol. 9, No. 3. pp. 336-351.
@article{712f87a9857649b78953999778626c45,
title = "Dealing with effect size multiplicity in systematic reviews and meta-analyses",
abstract = "Systematic reviews often encounter primary studies that report multiple effect sizes based on data from the same participants. These have the potential to introduce statistical dependency into the meta-analytic data set. In this paper, we provide a tutorial on dealing with effect size multiplicity within studies in the context of meta-analyses of intervention and association studies, recommending a three-step approach. The first step is to define the research question and consider the extent to which it mainly reflects interest in mean effect sizes (which we term a convergent approach) or an interest in exploring heterogeneity (which we term a divergent approach). A second step is to identify the types of multiplicities that appear in the initial database of effect sizes relevant to the research question, and we propose a categorization scheme to differentiate them. The third step is to select a strategy for dealing with each type of multiplicity. The researcher can choose between a reductionist meta-analytic approach, which is characterized by inclusion of a single effect size per study, and an integrative approach, characterized by inclusion of multiple effect sizes per study. We present an overview of available analysis strategies for dealing with effect size multiplicity within studies and provide recommendations intended to help researchers decide which strategy might be preferable in particular situations. Last, we offer caveats and cautions about addressing the challenges multiplicity poses for systematic reviews and meta-analyses.",
keywords = "Dependency, Effect size, Meta-analysis, Multiplicity, Systematic review",
author = "L{\'o}pez-L{\'o}pez, {Jos{\'e} A.} and Page, {Matthew J.} and Lipsey, {Mark W.} and Higgins, {Julian P.T.}",
year = "2018",
month = "9",
doi = "10.1002/jrsm.1310",
language = "English",
volume = "9",
pages = "336--351",
journal = "Research Synthesis Methods",
issn = "1759-2879",
publisher = "Wiley-Blackwell",
number = "3",

}

Dealing with effect size multiplicity in systematic reviews and meta-analyses. / López-López, José A.; Page, Matthew J.; Lipsey, Mark W.; Higgins, Julian P.T.

In: Research Synthesis Methods, Vol. 9, No. 3, 09.2018, p. 336-351.

Research output: Contribution to journalArticleOtherpeer-review

TY - JOUR

T1 - Dealing with effect size multiplicity in systematic reviews and meta-analyses

AU - López-López, José A.

AU - Page, Matthew J.

AU - Lipsey, Mark W.

AU - Higgins, Julian P.T.

PY - 2018/9

Y1 - 2018/9

N2 - Systematic reviews often encounter primary studies that report multiple effect sizes based on data from the same participants. These have the potential to introduce statistical dependency into the meta-analytic data set. In this paper, we provide a tutorial on dealing with effect size multiplicity within studies in the context of meta-analyses of intervention and association studies, recommending a three-step approach. The first step is to define the research question and consider the extent to which it mainly reflects interest in mean effect sizes (which we term a convergent approach) or an interest in exploring heterogeneity (which we term a divergent approach). A second step is to identify the types of multiplicities that appear in the initial database of effect sizes relevant to the research question, and we propose a categorization scheme to differentiate them. The third step is to select a strategy for dealing with each type of multiplicity. The researcher can choose between a reductionist meta-analytic approach, which is characterized by inclusion of a single effect size per study, and an integrative approach, characterized by inclusion of multiple effect sizes per study. We present an overview of available analysis strategies for dealing with effect size multiplicity within studies and provide recommendations intended to help researchers decide which strategy might be preferable in particular situations. Last, we offer caveats and cautions about addressing the challenges multiplicity poses for systematic reviews and meta-analyses.

AB - Systematic reviews often encounter primary studies that report multiple effect sizes based on data from the same participants. These have the potential to introduce statistical dependency into the meta-analytic data set. In this paper, we provide a tutorial on dealing with effect size multiplicity within studies in the context of meta-analyses of intervention and association studies, recommending a three-step approach. The first step is to define the research question and consider the extent to which it mainly reflects interest in mean effect sizes (which we term a convergent approach) or an interest in exploring heterogeneity (which we term a divergent approach). A second step is to identify the types of multiplicities that appear in the initial database of effect sizes relevant to the research question, and we propose a categorization scheme to differentiate them. The third step is to select a strategy for dealing with each type of multiplicity. The researcher can choose between a reductionist meta-analytic approach, which is characterized by inclusion of a single effect size per study, and an integrative approach, characterized by inclusion of multiple effect sizes per study. We present an overview of available analysis strategies for dealing with effect size multiplicity within studies and provide recommendations intended to help researchers decide which strategy might be preferable in particular situations. Last, we offer caveats and cautions about addressing the challenges multiplicity poses for systematic reviews and meta-analyses.

KW - Dependency

KW - Effect size

KW - Meta-analysis

KW - Multiplicity

KW - Systematic review

UR - http://www.scopus.com/inward/record.url?scp=85051034589&partnerID=8YFLogxK

U2 - 10.1002/jrsm.1310

DO - 10.1002/jrsm.1310

M3 - Article

VL - 9

SP - 336

EP - 351

JO - Research Synthesis Methods

JF - Research Synthesis Methods

SN - 1759-2879

IS - 3

ER -