Capturing domain expert perspectives in devising a rating scale for a health specific writing test: How close can we get?

The importance of input from occupational experts in defining valid criteria to assess performance on English for specific purposes (ESP) tests is widely acknowledged. However, few studies have described the process of collecting indigenous criteria and establishing their suitability for a language testing context. The paper reports on this process with specific reference to the writing sub-test of the Occupational English Test (OET) for overseas-trained health professionals. The OET writing task requires candidates, in the role of health professional, to draw on a set of written prompts to write a letter of referral to a colleague on an aspect of patient care. With the aim of expanding the test construct as reflected in the criteria, health professionals from hospitals were asked to judge the adequacy of patient records obtained from health settings. Based on their feedback, an elaborate checklist was developed to reflect the qualities of the written documents that they considered critical. The paper discusses the challenges involved in the process, including the inevitable construct shrinkage evident in the final version of the checklist indicators due to the constraints of the testing situation. The study has implications for our understanding of authenticity in ESP assessments.