Eliciting group judgements about replicability: a technical implementation of the IDEA Protocol

Eric Ross Pearson; Hannah Fraser; Martin Bush; Fallon Mody; Ivo Widjaja; Andy Head; David Peter Wilkinson; Richard Sinnott; Bonnie Claire Wintle; Mark Burgman; Fiona Fidler; Peter Vesk

Eliciting group judgements about replicability: a technical implementation of the IDEA Protocol

Eric Ross Pearson, Hannah Fraser, Martin Bush, Fallon Mody, Ivo Widjaja, Andy Head, David Peter Wilkinson, Richard Sinnott, Bonnie Claire Wintle, Mark Burgman, Fiona Fidler, Peter Vesk

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

1 Citation (Scopus)

Abstract

In recent years there has been increased interest in replicating prior research. One of the biggest challenges to assessing replicability is the cost in resources and time that it takes to repeat studies. Thus there is an impetus to develop rapid elicitation protocols that can, in a practical manner, estimate the likelihood that research findings will successfully replicate. We employ a novel implementation of the IDEA ('Investigate', 'Discuss', 'Estimate' and 'Aggregate) protocol, realised through the repliCATS platform. The repliCATS platform is designed to scalably elicit expert opinion about replicability of social and behavioural science research. The IDEA protocol provides a structured methodology for eliciting judgements and reasoning from groups. This paper describes the repliCATS platform as a multi-user cloud-based software platform featuring (1) a technical implementation of the IDEA protocol for eliciting expert opinion on research replicability, (2) capture of consent and demographic data, (3) on-line training on replication concepts, and (4) exporting of completed judgements. The platform has, to date, evaluated 3432 social and behavioural science research claims from 637 participants.

Original language	English
Title of host publication	Proceedings of the 54th Annual Hawaii International Conference on System Sciences
Editors	Tung X. Bui
Place of Publication	New York NY USA
Publisher	IEEE, Institute of Electrical and Electronics Engineers
Pages	461-470
Number of pages	10
ISBN (Electronic)	9780998133140
Publication status	Published - 2021
Externally published	Yes
Event	Hawaii International Conference on System Sciences 2021 - Online, Virtual, Online, United States of America Duration: 4 Jan 2021 → 8 Jan 2021 Conference number: 54th https://scholarspace.manoa.hawaii.edu/handle/10125/72112 (Proceedings/Website)

Publication series

Name	Proceedings of the Annual Hawaii International Conference on System Sciences
Publisher	IEEE, Institute of Electrical and Electronics Engineers
Volume	2020-January
ISSN (Print)	1530-1605

Conference

Conference	Hawaii International Conference on System Sciences 2021
Abbreviated title	HICSS 2021
Country/Territory	United States of America
City	Virtual, Online
Period	4/01/21 → 8/01/21
Internet address	https://scholarspace.manoa.hawaii.edu/handle/10125/72112 (Proceedings/Website)

Keywords

Expert elicitation
IDEA protocol
Replicability
RepliCATS

Access to Document

375245145-oaFinal published version, 1.39 MB

Cite this

Pearson, E. R., Fraser, H., Bush, M., Mody, F., Widjaja, I., Head, A., Wilkinson, D. P., Sinnott, R., Wintle, B. C., Burgman, M., Fidler, F., & Vesk, P. (2021). Eliciting group judgements about replicability: a technical implementation of the IDEA Protocol. In T. X. Bui (Ed.), Proceedings of the 54th Annual Hawaii International Conference on System Sciences (pp. 461-470). (Proceedings of the Annual Hawaii International Conference on System Sciences; Vol. 2020-January). IEEE, Institute of Electrical and Electronics Engineers.

Pearson, Eric Ross ; Fraser, Hannah ; Bush, Martin et al. / Eliciting group judgements about replicability : a technical implementation of the IDEA Protocol. Proceedings of the 54th Annual Hawaii International Conference on System Sciences. editor / Tung X. Bui. New York NY USA : IEEE, Institute of Electrical and Electronics Engineers, 2021. pp. 461-470 (Proceedings of the Annual Hawaii International Conference on System Sciences).

@inproceedings{cd34a853aa4e4a86a9dcd4cea9008720,

title = "Eliciting group judgements about replicability: a technical implementation of the IDEA Protocol",

abstract = "In recent years there has been increased interest in replicating prior research. One of the biggest challenges to assessing replicability is the cost in resources and time that it takes to repeat studies. Thus there is an impetus to develop rapid elicitation protocols that can, in a practical manner, estimate the likelihood that research findings will successfully replicate. We employ a novel implementation of the IDEA ('Investigate', 'Discuss', 'Estimate' and 'Aggregate) protocol, realised through the repliCATS platform. The repliCATS platform is designed to scalably elicit expert opinion about replicability of social and behavioural science research. The IDEA protocol provides a structured methodology for eliciting judgements and reasoning from groups. This paper describes the repliCATS platform as a multi-user cloud-based software platform featuring (1) a technical implementation of the IDEA protocol for eliciting expert opinion on research replicability, (2) capture of consent and demographic data, (3) on-line training on replication concepts, and (4) exporting of completed judgements. The platform has, to date, evaluated 3432 social and behavioural science research claims from 637 participants.",

keywords = "Expert elicitation, IDEA protocol, Replicability, RepliCATS",

author = "Pearson, {Eric Ross} and Hannah Fraser and Martin Bush and Fallon Mody and Ivo Widjaja and Andy Head and Wilkinson, {David Peter} and Richard Sinnott and Wintle, {Bonnie Claire} and Mark Burgman and Fiona Fidler and Peter Vesk",

note = "Funding Information: This research is based upon work funded by the Defence Advanced Research Projects Activity (DARPA), under Contract HR001118S0047 as part of the SCORE program. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. Funding Information: The repliCATS platform described here was developed as part of the SCORE (Systematizing Confidence in Open Research and Evidence) program funded by DARPA (Defence Advanced Research Projects Agency)1. The overall goal of the SCORE project is to create automated tools to assign {\textquoteleft}confidence scores{\textquoteright} to the replicability of research claims within the social and behavioural science literature. The SCORE program consists of three Technical Aspects, comprising multiple independent teams (known as {\textquoteleft}performers{\textquoteright}). The role of the repliCATS project as part of the second Technical Aspect (TA2) is to elicit human assessments of replicability for 3000 research claims. Publisher Copyright: {\textcopyright} 2021 IEEE Computer Society. All rights reserved.; Hawaii International Conference on System Sciences 2021, HICSS 2021 ; Conference date: 04-01-2021 Through 08-01-2021",

year = "2021",

language = "English",

series = "Proceedings of the Annual Hawaii International Conference on System Sciences",

publisher = "IEEE, Institute of Electrical and Electronics Engineers",

pages = "461--470",

editor = "Bui, {Tung X.}",

booktitle = "Proceedings of the 54th Annual Hawaii International Conference on System Sciences",

address = "United States of America",

url = "https://scholarspace.manoa.hawaii.edu/handle/10125/72112",

}

Pearson, ER, Fraser, H, Bush, M, Mody, F, Widjaja, I, Head, A, Wilkinson, DP, Sinnott, R, Wintle, BC, Burgman, M, Fidler, F & Vesk, P 2021, Eliciting group judgements about replicability: a technical implementation of the IDEA Protocol. in TX Bui (ed.), Proceedings of the 54th Annual Hawaii International Conference on System Sciences. Proceedings of the Annual Hawaii International Conference on System Sciences, vol. 2020-January, IEEE, Institute of Electrical and Electronics Engineers, New York NY USA, pp. 461-470, Hawaii International Conference on System Sciences 2021, Virtual, Online, Hawaii, United States of America, 4/01/21.

Eliciting group judgements about replicability: a technical implementation of the IDEA Protocol. / Pearson, Eric Ross; Fraser, Hannah; Bush, Martin et al.
Proceedings of the 54th Annual Hawaii International Conference on System Sciences. ed. / Tung X. Bui. New York NY USA: IEEE, Institute of Electrical and Electronics Engineers, 2021. p. 461-470 (Proceedings of the Annual Hawaii International Conference on System Sciences; Vol. 2020-January).

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - Eliciting group judgements about replicability

T2 - Hawaii International Conference on System Sciences 2021

AU - Pearson, Eric Ross

AU - Fraser, Hannah

AU - Bush, Martin

AU - Mody, Fallon

AU - Widjaja, Ivo

AU - Head, Andy

AU - Wilkinson, David Peter

AU - Sinnott, Richard

AU - Wintle, Bonnie Claire

AU - Burgman, Mark

AU - Fidler, Fiona

AU - Vesk, Peter

N1 - Conference code: 54th

PY - 2021

Y1 - 2021

N2 - In recent years there has been increased interest in replicating prior research. One of the biggest challenges to assessing replicability is the cost in resources and time that it takes to repeat studies. Thus there is an impetus to develop rapid elicitation protocols that can, in a practical manner, estimate the likelihood that research findings will successfully replicate. We employ a novel implementation of the IDEA ('Investigate', 'Discuss', 'Estimate' and 'Aggregate) protocol, realised through the repliCATS platform. The repliCATS platform is designed to scalably elicit expert opinion about replicability of social and behavioural science research. The IDEA protocol provides a structured methodology for eliciting judgements and reasoning from groups. This paper describes the repliCATS platform as a multi-user cloud-based software platform featuring (1) a technical implementation of the IDEA protocol for eliciting expert opinion on research replicability, (2) capture of consent and demographic data, (3) on-line training on replication concepts, and (4) exporting of completed judgements. The platform has, to date, evaluated 3432 social and behavioural science research claims from 637 participants.

AB - In recent years there has been increased interest in replicating prior research. One of the biggest challenges to assessing replicability is the cost in resources and time that it takes to repeat studies. Thus there is an impetus to develop rapid elicitation protocols that can, in a practical manner, estimate the likelihood that research findings will successfully replicate. We employ a novel implementation of the IDEA ('Investigate', 'Discuss', 'Estimate' and 'Aggregate) protocol, realised through the repliCATS platform. The repliCATS platform is designed to scalably elicit expert opinion about replicability of social and behavioural science research. The IDEA protocol provides a structured methodology for eliciting judgements and reasoning from groups. This paper describes the repliCATS platform as a multi-user cloud-based software platform featuring (1) a technical implementation of the IDEA protocol for eliciting expert opinion on research replicability, (2) capture of consent and demographic data, (3) on-line training on replication concepts, and (4) exporting of completed judgements. The platform has, to date, evaluated 3432 social and behavioural science research claims from 637 participants.

KW - Expert elicitation

KW - IDEA protocol

KW - Replicability

KW - RepliCATS

UR - http://www.scopus.com/inward/record.url?scp=85108347219&partnerID=8YFLogxK

M3 - Conference Paper

AN - SCOPUS:85108347219

T3 - Proceedings of the Annual Hawaii International Conference on System Sciences

SP - 461

EP - 470

BT - Proceedings of the 54th Annual Hawaii International Conference on System Sciences

A2 - Bui, Tung X.

PB - IEEE, Institute of Electrical and Electronics Engineers

CY - New York NY USA

Y2 - 4 January 2021 through 8 January 2021

ER -

Pearson ER, Fraser H, Bush M, Mody F, Widjaja I, Head A et al. Eliciting group judgements about replicability: a technical implementation of the IDEA Protocol. In Bui TX, editor, Proceedings of the 54th Annual Hawaii International Conference on System Sciences. New York NY USA: IEEE, Institute of Electrical and Electronics Engineers. 2021. p. 461-470. (Proceedings of the Annual Hawaii International Conference on System Sciences).

Eliciting group judgements about replicability: a technical implementation of the IDEA Protocol

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Cite this