Development and implementation of a Situational Judgement Test Scores (SJT) across five health disciplines

Title: Development and implementation of a Situational Judgement Test Scores (SJT) across five health disciplines Authors: Irene Lichtwark, Samuel Henry, Loretta Garvey, Suzanne Metcalf, Andrea Bryce, Caroline Wright, Margaret Hay. Presenter: Irene Lichtwark Introduction: Situational Judgement Tests (SJTs) are widely used for the medical student selection, and also for selection into post graduate medical training. Surprisingly, given the prevalence of SJTs in medical selection, the tool has not been widely used to select students into the allied health professions. SJTs are complex to develop, requiring significant expertise (Patterson et al., 2016), and financial investment, which may account for their limited uptake in allied health student selection. This study describes the development and implementation of SJTs across five health disciplines in Australia. These were Nursing, Dietetics, Paramedics, Radiation Therapy, and Medicine. Three of the disciplines have implemented the developed SJT for validity determination, with the other two disciplines implementing in early 2017. Comparative available results to date are presented. Methods: Over two half-day workshops, practitioners across the five health disciplines developed 33 SJT scenarios, each with six associated rating statements across four domains (Empathy, Integrity, Collaboration, and Resilience). Seven items were deleted due to subjectivity of scenario interpretation. Of the 24 remaining items, seven were contextualised for each discipline and 17 were generic in the situation, but contextualised for the depicted character (e.g., ‘Shantha, a dietetic student …). Contextualisation and scoring concordance were undertaken by a second group of representatives from each discipline. The 17 generic and seven contextualised SJT items were programmed into QualtricsTM and administered to the disciplines of Nursing (n=309), Dietetics (n=77), and Radiation Therapy (n=12) for psychometric validation (N= 398). Item quality, difficulty and test level reliability were investigated for each discipline separately. Kruskal-Wallis and Mann-Whitney tests were used to explore differences in total and domain scores between nursing, dietetic, and Radiation Therapy students. Results: The 17 SJT items correlated (r=0.2) and achieved good reliability (alpha=0.85). Significant differences in mean rank were found between Nursing and Dietetics students (U=7655.5, p= .001) with nursing scoring higher. The overall median scores varied little between health professions on SJT total score. Significant differences were found for Empathy, Collaboration and Integrity (χ2 (3) = 12.60, p= 0.006, χ2 (3) = 10.31, p = 0.016 and χ2 (3) = 13.46, p= 0.004, respectively), with a Bonferroni correction of alpha = 0.017. Radiation Therapy students obtained the highest median score for Integrity. Nursing students obtained higher median scores for Empathy and Collaboration. No significant differences between the disciplines were found in Resilience. Boxplots revealed outliers at the bottom rather than the top across disciplines. Discussion: This study confirms that collaboration across health disciplines to develop SJT items, including generic items for use across disciplines, is possible and practical. This is the first study to describe a pattern of SJT scores across diverse health disciplines. Results indicate little difference across disciplines in overall score, but differences in three of the four test domains across disciplines were evident. Our findings indicate the value of domain scores for selection decisions, and the importance of including a broad range of domains in the test. Interestingly, resilience showed no difference across disciplines. That outliers occur almost exclusively at the lower end of the scoring scale indicates that SJT can differentiate between low scoring applicants and can be used to screen out unsuitable applicants across disciplines. Reference Patterson, F., Knight, A., Dowell, J., Nicholson, S., Cousans, F., & Cleland, J. (2016). How effective are selection methods in medical education? A systematic review. Medical education, 50(1), 36-60.