Inter-rater reliability of three adult handwriting legibility instruments

Background/aim: Legibility is important for functional, handwritten communication. Deficits in legibility can impair occupational performance following stroke or trauma. Few instruments are available to assess adult handwriting legibility during rehabilitation. The aim of this study was to compare inter-rater reliability of a new four-point legibility rating instrument with two existing instruments, and describe scale structure and item difficulty of each instrument. Methods: Three trained raters scored 60 handwriting samples using: (i) a Four-Point Scale (FPS); (ii) the modified Evaluation Tool of Children's Handwriting (mETCH); and (iii) the new modified FPS. Rater concordance and exact agreement were investigated using the intra-class correlation coefficient (ICC_3,1), multi-rater kappa (κ) and Krippendorff's alpha (α). Cronbach's alpha was calculated to examine internal consistency reliability, and Rasch modelling was used to examine scale structure and item difficulty. Results: Rater concordance for the FPS was fair (ICC_3,1 = 0.37); exact agreement was poor (κ = 0.19; α = 0.19). Rater concordance for the two mETCH subtests was fair to moderate (ICC_3,1 = 0.39-0.50), but with no exact agreement (κ < 0.00). Rater concordance for three subtests of the modified FPS was slight to moderate (ICC_3,1 = 0.16-0.51); exact agreement ranged from nil to fair (κ = -0.06-0.30; α = -0.05-0.30). Rasch modelling confirmed internal consistency of instruments, but was low between-rater consistency (rater severity variability). Conclusions: A reliable instrument for measuring change over time remains elusive. In the meantime, these instruments can be used by individual clinicians to diagnose and rate legibility.