Abstract
Background: Although a thoughtfully designed Objective Structured Clinical Examination(OSCE) is a robust and valid assessment tool, debate exists about the effectiveness of OSCEs to authentically mirror real-life scenarios. Limited studies have explored the extrapolation inference in Kane Validity’s Framework.
Summary of work: A sequential mixed methods approach was used. Mystery shoppers visited pharmacy students on their community pharmacy placement and simulated the same case scenario students were given a recent infectious diseases OSCE. Students were marked with the same rubrics and these marks were compared with their OSCE score. The mystery shopping visit was then revealed to all the students and all students were asked to participate in a semi structured interview.
Results: Overall, 92 mystery shopper (Work Based Assessment (WBA)) visits with students were conducted and 36 follow-up interviews were completed. The median WBA score was39.2% lower compared to the OSCE score (p < 0.001). Interviews revealed students knew they did not perform as well in the WBA compared to their OSCE, but reflected that they still need OSCEs to prepare them to manage a real-life patient.
Discussion: Many students related their performance to how they perceived their role in OSCEs versus WBAs, and that OSCEs allowed them more autonomy to manage the patient as oppose to an unfamiliar workplace.
Conclusions: OSCEs performance did not correlate to real life performance in this instance; however, students still valued having an OSCE before a WBA.
Take-home messages: Students scored lower on placement than in OSCEs even though they reflected that it is easier to manage a patient in real-life; outlining challenges to replicate a real life pharmacy situation in an OSCE, and challenges to testing extrapolation. Whilst OSCEs are useful for testing process type skills, clinical problem solving may be best assessed in a workplace environment.
Summary of work: A sequential mixed methods approach was used. Mystery shoppers visited pharmacy students on their community pharmacy placement and simulated the same case scenario students were given a recent infectious diseases OSCE. Students were marked with the same rubrics and these marks were compared with their OSCE score. The mystery shopping visit was then revealed to all the students and all students were asked to participate in a semi structured interview.
Results: Overall, 92 mystery shopper (Work Based Assessment (WBA)) visits with students were conducted and 36 follow-up interviews were completed. The median WBA score was39.2% lower compared to the OSCE score (p < 0.001). Interviews revealed students knew they did not perform as well in the WBA compared to their OSCE, but reflected that they still need OSCEs to prepare them to manage a real-life patient.
Discussion: Many students related their performance to how they perceived their role in OSCEs versus WBAs, and that OSCEs allowed them more autonomy to manage the patient as oppose to an unfamiliar workplace.
Conclusions: OSCEs performance did not correlate to real life performance in this instance; however, students still valued having an OSCE before a WBA.
Take-home messages: Students scored lower on placement than in OSCEs even though they reflected that it is easier to manage a patient in real-life; outlining challenges to replicate a real life pharmacy situation in an OSCE, and challenges to testing extrapolation. Whilst OSCEs are useful for testing process type skills, clinical problem solving may be best assessed in a workplace environment.
| Original language | English |
|---|---|
| Number of pages | 1 |
| Publication status | Published - 2024 |
| Event | Ottawa Conference 2024: Assessment: Near and far - Melbourne Convention and Exhibition Centre, Melbourne, Australia Duration: 24 Feb 2024 → 28 Feb 2024 https://ottawa2024.au/Microsite/content.html (Abstracts) http://ottawa2024.au/ https://ottawa2024.au/program/ |
Conference
| Conference | Ottawa Conference 2024 |
|---|---|
| Abbreviated title | Ottawa 2024 |
| Country/Territory | Australia |
| City | Melbourne |
| Period | 24/02/24 → 28/02/24 |
| Other | Ottawa Conference on the Assessment of Competence in Medical and the Health Care Professions 2024 |
| Internet address |
