Harnessing Generative AI for Objective Structured Clinical Examinations Preparation in Pharmacy Education

Introduction. Objective Structured Clinical Examinations (OSCEs) are widely utilised in pharmacy education to assess readiness for practice. There is demand for individualised simulations that allow students to prepare for OSCEs through repeated practice. These simulations can complement traditional face-to-face learning, providing additional tools and creating more opportunities for practice, while still offering individualised feedback and overcoming the logistical and budgetary challenges associated with traditional face-to-face formative OSCEs.

Aims. To evaluate the usability and acceptability of ATLAS (Authentic Teaching & Learning Application Simulations) as a tool for OSCE preparation. This technology harnesses cutting-edge Large Language Models (e.g. OpenAI’s GPT-4) to provide students with real-time, tailored feedback on both clinical and communication skills.

Methods. A quasi-experimental study was conducted across two pharmacy units studied at Monash University. Data encompassing ATLAS attempt counts, OSCE scores, and tool perceptions were systematically collected and analysed. The study evaluated the influence of ATLAS on both domestic and international students’ OSCE performance. Additionally, a content analysis of qualitative feedback pinpointed the advantages and challenges of integrating generative AI using ATLAS into OSCE preparation.

Results. Out of the 328 students who participated, 263 used ATLAS. Median OSCE scores improved with increased ATLAS practice, from 0.80 with no attempts to 0.90 after six attempts. OLS regression analysis showed a significant impact of ATLAS usage on OSCE marks (p=0.01), with an estimated increase of 0.09 in scores from zero to six conversations. Survey feedback indicated students valued the simulations for OSCE preparation.

Discussion. The ATLAS platform’s contribution to OSCE preparation was positively received, with students indicating that these simulations were realistic. The tool proved to be effective as students obtained an improvement in their clinical and communication skills. This highlights ATLAS’s value as a supplementary educational tool, fostering enhanced patient interaction competencies. However, the lag in responding to the student’s input and preference for in-person interactions revealed the limitations of the current technology and highlighted its potential to complement rather than replace in-person simulations.