TY - JOUR
T1 - Statistical models for repeated categorical ratings
T2 - the R package rater
AU - Pullin, Jeffrey M.
AU - Gurrin, Lyle C.
AU - Vukcevic, Damjan
N1 - Funding Information:
We would like to thank Bob Carpenter for many helpful suggestions, and David Whitelaw for his flexibility in allowing the first author to work on this paper while employed at the Australian Institute of Heath and Welfare. Thanks also to Lars Mølgaard Saxhaug for his code contributions to rater.
Publisher Copyright:
© (2023), (Technische Universitaet Wien). All Rights Reserved.
PY - 2023
Y1 - 2023
N2 - A common problem in many disciplines is the need to assign a set of items into categories or classes with known labels. This is often done by one or more expert raters, or sometimes by an automated process. If these assignments or ‘ratings’ are difficult to make accurately, a common tactic is to repeat them by different raters, or even by the same rater multiple times on different occasions. We present an R package rater, available on CRAN, that implements Bayesian versions of several statistical models for analysis of repeated categorical rating data. Inference is possible for the true underlying (latent) class of each item, as well as the accuracy of each rater. The models are extensions of, and include, the Dawid–Skene model, and we implemented them using the Stan probabilistic programming language. We illustrate the use of rater through a few examples. We also discuss in detail the techniques of marginalisation and conditioning, which are necessary for these models but also apply more generally to other models implemented in Stan.
AB - A common problem in many disciplines is the need to assign a set of items into categories or classes with known labels. This is often done by one or more expert raters, or sometimes by an automated process. If these assignments or ‘ratings’ are difficult to make accurately, a common tactic is to repeat them by different raters, or even by the same rater multiple times on different occasions. We present an R package rater, available on CRAN, that implements Bayesian versions of several statistical models for analysis of repeated categorical rating data. Inference is possible for the true underlying (latent) class of each item, as well as the accuracy of each rater. The models are extensions of, and include, the Dawid–Skene model, and we implemented them using the Stan probabilistic programming language. We illustrate the use of rater through a few examples. We also discuss in detail the techniques of marginalisation and conditioning, which are necessary for these models but also apply more generally to other models implemented in Stan.
UR - http://www.scopus.com/inward/record.url?scp=85184772202&partnerID=8YFLogxK
U2 - 10.32614/RJ-2023-064
DO - 10.32614/RJ-2023-064
M3 - Article
AN - SCOPUS:85184772202
SN - 2073-4859
VL - 15
SP - 93
EP - 118
JO - The R Journal
JF - The R Journal
IS - 3
ER -