Comparing correlated kappas by resampling: Is one level of agreement significantly different from another?

Dean P. McKenzie, Andrew J. Mackinnon, Normand Péladeau, Patrick Onghena, Peter C. Bruce, David M. Clarke, Susan Harrigan, Patrick D. McGorry

Research output: Contribution to journalArticleResearchpeer-review

77 Citations (Scopus)


Researchers comparing diagnostic systems or screening tests frequently need to compare indices of agreement such as the kappa coefficient. While asymptotic methods exist for comparing kappas derived from independent samples, no satisfactory approach exists for comparing kappas derived from the same or related samples. Resampling methods for comparing kappa coefficients obtained from the same sample are presented. Easily undertaken using readily available software, these methods are illustrated by application to a small sample of psychiatric data, as well as to several thousand samples of simulated data. An acceptable type I error rate was exhibited. Resampling techniques - easily implemented and making few assumptions - deserve wider application in psychiatric research.

Original languageEnglish
Pages (from-to)483-492
Number of pages10
JournalJournal of Psychiatric Research
Issue number6
Publication statusPublished - 1 Nov 1996

Cite this