The binary replicate test: determining the sensitivity of CSCL models to coding error

Brendan R. Eagan; Zachari Swiecki; Cayley Farrell; David Williamson Shaffer

The binary replicate test: determining the sensitivity of CSCL models to coding error

Brendan R. Eagan, Zachari Swiecki, Cayley Farrell, David Williamson Shaffer

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

Abstract

The process of labeling, categorizing, or otherwise annotating data—or coding in the computer-supported collaborative learning (CSCL) literature—is a fundamental process in CSCL research. It is the process by which researchers identify salient properties about segments of CSCL data: What they are, what they contain, or what they mean. Coding, like all processes in research, is subject to error. To reduce the potential impact of coding error, CSCL researchers typically measure inter-rater reliability (IRR). However, there is no extant method to determine what level of IRR would invalidate a CSCL result or model. One way of assessing the potential impact of such inaccuracies is by conducting sensitivity analyses, which measure the level of error that would need to be present in the data to invalidate a given inference. This paper introduces a new method for conducting sensitivity analyses in CSCL: The Binary Replicate Test.

Original language	English
Title of host publication	13th International Conference on Computer Supported Collaborative Learning
Subtitle of host publication	Conference Proceedings
Editors	Kristine Lund, Gerald P. Niccolai, Elise Lavoue, Cindy Hmelo-Silver, Gahgene Gweon, Michael Baker
Place of Publication	Lyon France
Publisher	International Society of the Learning Sciences (ISLS)
Pages	328-335
Number of pages	8
ISBN (Electronic)	9781732467231
Publication status	Published - 2019
Externally published	Yes
Event	Computer Supported Collaborative Learning 2019 - Lyon, France Duration: 17 Jun 2019 → 21 Jun 2019 Conference number: 13th https://www.cscl2019.com/ https://www.isls.org/cscl/2019/www.cscl2019.com/index.html (Proceedings)

Publication series

Name	Computer-Supported Collaborative Learning Conference, CSCL
Publisher	International Society of the Learning Sciences, Inc.
Volume	1
ISSN (Electronic)	1573-4552

Conference

Conference	Computer Supported Collaborative Learning 2019
Abbreviated title	CSCL 2019
Country	France
City	Lyon
Period	17/06/19 → 21/06/19
Internet address	https://www.cscl2019.com/ https://www.isls.org/cscl/2019/www.cscl2019.com/index.html (Proceedings)

Access to Document

330069035-oaFinal published version, 1.09 MB

Cite this

Eagan, B. R., Swiecki, Z., Farrell, C., & Shaffer, D. W. (2019). The binary replicate test: determining the sensitivity of CSCL models to coding error. In K. Lund, G. P. Niccolai, E. Lavoue, C. Hmelo-Silver, G. Gweon, & M. Baker (Eds.), 13th International Conference on Computer Supported Collaborative Learning: Conference Proceedings (pp. 328-335). (Computer-Supported Collaborative Learning Conference, CSCL; Vol. 1). International Society of the Learning Sciences (ISLS). https://www.isls.org/cscl/2019/www.cscl2019.com/index.html

Eagan, Brendan R. ; Swiecki, Zachari ; Farrell, Cayley ; Shaffer, David Williamson. / The binary replicate test : determining the sensitivity of CSCL models to coding error. 13th International Conference on Computer Supported Collaborative Learning: Conference Proceedings . editor / Kristine Lund ; Gerald P. Niccolai ; Elise Lavoue ; Cindy Hmelo-Silver ; Gahgene Gweon ; Michael Baker. Lyon France : International Society of the Learning Sciences (ISLS), 2019. pp. 328-335 (Computer-Supported Collaborative Learning Conference, CSCL).

@inproceedings{8a8ae73c9ddc49b4bfe9216f9fb9d24f,

title = "The binary replicate test: determining the sensitivity of CSCL models to coding error",

abstract = "The process of labeling, categorizing, or otherwise annotating data—or coding in the computer-supported collaborative learning (CSCL) literature—is a fundamental process in CSCL research. It is the process by which researchers identify salient properties about segments of CSCL data: What they are, what they contain, or what they mean. Coding, like all processes in research, is subject to error. To reduce the potential impact of coding error, CSCL researchers typically measure inter-rater reliability (IRR). However, there is no extant method to determine what level of IRR would invalidate a CSCL result or model. One way of assessing the potential impact of such inaccuracies is by conducting sensitivity analyses, which measure the level of error that would need to be present in the data to invalidate a given inference. This paper introduces a new method for conducting sensitivity analyses in CSCL: The Binary Replicate Test.",

author = "Eagan, {Brendan R.} and Zachari Swiecki and Cayley Farrell and Shaffer, {David Williamson}",

year = "2019",

language = "English",

series = "Computer-Supported Collaborative Learning Conference, CSCL",

publisher = "International Society of the Learning Sciences (ISLS)",

pages = "328--335",

editor = "Kristine Lund and Niccolai, {Gerald P.} and Elise Lavoue and Cindy Hmelo-Silver and Gahgene Gweon and Michael Baker",

booktitle = "13th International Conference on Computer Supported Collaborative Learning",

note = "Computer Supported Collaborative Learning 2019, CSCL 2019 ; Conference date: 17-06-2019 Through 21-06-2019",

url = "https://www.cscl2019.com/, https://www.isls.org/cscl/2019/www.cscl2019.com/index.html",

}

Eagan, BR, Swiecki, Z, Farrell, C & Shaffer, DW 2019, The binary replicate test: determining the sensitivity of CSCL models to coding error. in K Lund, GP Niccolai, E Lavoue, C Hmelo-Silver, G Gweon & M Baker (eds), 13th International Conference on Computer Supported Collaborative Learning: Conference Proceedings . Computer-Supported Collaborative Learning Conference, CSCL, vol. 1, International Society of the Learning Sciences (ISLS), Lyon France, pp. 328-335, Computer Supported Collaborative Learning 2019, Lyon, France, 17/06/19. <https://www.isls.org/cscl/2019/www.cscl2019.com/index.html>

The binary replicate test : determining the sensitivity of CSCL models to coding error. / Eagan, Brendan R.; Swiecki, Zachari; Farrell, Cayley; Shaffer, David Williamson.

13th International Conference on Computer Supported Collaborative Learning: Conference Proceedings . ed. / Kristine Lund; Gerald P. Niccolai; Elise Lavoue; Cindy Hmelo-Silver; Gahgene Gweon; Michael Baker. Lyon France : International Society of the Learning Sciences (ISLS), 2019. p. 328-335 (Computer-Supported Collaborative Learning Conference, CSCL; Vol. 1).

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - The binary replicate test

T2 - Computer Supported Collaborative Learning 2019

AU - Eagan, Brendan R.

AU - Swiecki, Zachari

AU - Farrell, Cayley

AU - Shaffer, David Williamson

N1 - Conference code: 13th

PY - 2019

Y1 - 2019

N2 - The process of labeling, categorizing, or otherwise annotating data—or coding in the computer-supported collaborative learning (CSCL) literature—is a fundamental process in CSCL research. It is the process by which researchers identify salient properties about segments of CSCL data: What they are, what they contain, or what they mean. Coding, like all processes in research, is subject to error. To reduce the potential impact of coding error, CSCL researchers typically measure inter-rater reliability (IRR). However, there is no extant method to determine what level of IRR would invalidate a CSCL result or model. One way of assessing the potential impact of such inaccuracies is by conducting sensitivity analyses, which measure the level of error that would need to be present in the data to invalidate a given inference. This paper introduces a new method for conducting sensitivity analyses in CSCL: The Binary Replicate Test.

AB - The process of labeling, categorizing, or otherwise annotating data—or coding in the computer-supported collaborative learning (CSCL) literature—is a fundamental process in CSCL research. It is the process by which researchers identify salient properties about segments of CSCL data: What they are, what they contain, or what they mean. Coding, like all processes in research, is subject to error. To reduce the potential impact of coding error, CSCL researchers typically measure inter-rater reliability (IRR). However, there is no extant method to determine what level of IRR would invalidate a CSCL result or model. One way of assessing the potential impact of such inaccuracies is by conducting sensitivity analyses, which measure the level of error that would need to be present in the data to invalidate a given inference. This paper introduces a new method for conducting sensitivity analyses in CSCL: The Binary Replicate Test.

UR - http://www.scopus.com/inward/record.url?scp=85073394584&partnerID=8YFLogxK

M3 - Conference Paper

AN - SCOPUS:85073394584

T3 - Computer-Supported Collaborative Learning Conference, CSCL

SP - 328

EP - 335

BT - 13th International Conference on Computer Supported Collaborative Learning

A2 - Lund, Kristine

A2 - Niccolai, Gerald P.

A2 - Lavoue, Elise

A2 - Hmelo-Silver, Cindy

A2 - Gweon, Gahgene

A2 - Baker, Michael

PB - International Society of the Learning Sciences (ISLS)

CY - Lyon France

Y2 - 17 June 2019 through 21 June 2019

ER -

Eagan BR, Swiecki Z, Farrell C, Shaffer DW. The binary replicate test: determining the sensitivity of CSCL models to coding error. In Lund K, Niccolai GP, Lavoue E, Hmelo-Silver C, Gweon G, Baker M, editors, 13th International Conference on Computer Supported Collaborative Learning: Conference Proceedings . Lyon France: International Society of the Learning Sciences (ISLS). 2019. p. 328-335. (Computer-Supported Collaborative Learning Conference, CSCL).