The binary replicate test: determining the sensitivity of CSCL models to coding error

Brendan R. Eagan, Zachari Swiecki, Cayley Farrell, David Williamson Shaffer

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

3 Citations (Scopus)

Abstract

The process of labeling, categorizing, or otherwise annotating data—or coding in the computer-supported collaborative learning (CSCL) literature—is a fundamental process in CSCL research. It is the process by which researchers identify salient properties about segments of CSCL data: What they are, what they contain, or what they mean. Coding, like all processes in research, is subject to error. To reduce the potential impact of coding error, CSCL researchers typically measure inter-rater reliability (IRR). However, there is no extant method to determine what level of IRR would invalidate a CSCL result or model. One way of assessing the potential impact of such inaccuracies is by conducting sensitivity analyses, which measure the level of error that would need to be present in the data to invalidate a given inference. This paper introduces a new method for conducting sensitivity analyses in CSCL: The Binary Replicate Test.

Original languageEnglish
Title of host publication13th International Conference on Computer Supported Collaborative Learning
Subtitle of host publicationConference Proceedings
EditorsKristine Lund, Gerald P. Niccolai, Elise Lavoue, Cindy Hmelo-Silver, Gahgene Gweon, Michael Baker
Place of PublicationLyon France
PublisherInternational Society of the Learning Sciences (ISLS)
Pages328-335
Number of pages8
ISBN (Electronic)9781732467231
Publication statusPublished - 2019
Externally publishedYes
EventComputer Supported Collaborative Learning 2019 - Lyon, France
Duration: 17 Jun 201921 Jun 2019
Conference number: 13th
https://www.cscl2019.com/
https://www.isls.org/cscl/2019/www.cscl2019.com/index.html (Proceedings)

Publication series

NameComputer-Supported Collaborative Learning Conference, CSCL
PublisherInternational Society of the Learning Sciences, Inc.
Volume1
ISSN (Electronic)1573-4552

Conference

ConferenceComputer Supported Collaborative Learning 2019
Abbreviated titleCSCL 2019
CountryFrance
CityLyon
Period17/06/1921/06/19
Internet address

Cite this