Making robust assessments of specialist trainees' workplace performance

Jennifer M Weller, D. J. Castanelli, B. Jolly, Yan Chen

Research output: Contribution to journalArticleResearchpeer-review

Abstract

Background. Workplace-based assessments should provide a reliable measure of trainee performance, but have met with mixed success. We proposed that using an entrustability scale, where supervisors scored trainees on the level of supervision required for the case would improve the utility of compulsory mini-clinical evaluation exercise (CEX) assessments in a large anaesthesia training program. Methods. We analysed mini-CEX scores from all Australian and New Zealand College of Anaesthetists trainees submitted to an online database over a 12-month period. Supervisors' scores were adjusted for the expected supervision requirement for the case for trainees at different stages of training. We used generalisability theory to determine score reliability. Results. 7808 assessments were available for analysis. Supervision requirements decreased significantly (P < 0.05) with increased duration and level of training, supporting validity. We found moderate reliability (G > 0.7) with a feasible number of assessments. Adjusting scores against the expected supervision requirement considerably improved reliability, with G > 0.8 achieved with only nine assessments. Three per cent of trainees generated average mini-CEX scores below the expected standard. Conclusions. Using an entrustment scoring system, where supervisors score trainees on the level of supervision required, mini-CEX scores demonstrated moderate reliability within a feasible number of assessments, and evidence of validity. When scores were adjusted against an expected standard, underperforming trainees could be identified, and reliability much improved. Taken together with other evidence on trainee ability, the mini-CEX is of sufficient reliability for inclusion in high stakes decisions on trainee progression towards independent specialist practice.

Original languageEnglish
Pages (from-to)207-214
Number of pages8
JournalBritish Journal of Anaesthesia
Volume118
Issue number2
DOIs
Publication statusPublished - 1 Feb 2017

Keywords

  • Educational assessment
  • Educational measurement
  • Graduate
  • Medical education
  • Reliability
  • Workplace

Cite this

Weller, Jennifer M ; Castanelli, D. J. ; Jolly, B. ; Chen, Yan. / Making robust assessments of specialist trainees' workplace performance. In: British Journal of Anaesthesia. 2017 ; Vol. 118, No. 2. pp. 207-214.
@article{ae02cf0bfe3b4abe9a670e145de46922,
title = "Making robust assessments of specialist trainees' workplace performance",
abstract = "Background. Workplace-based assessments should provide a reliable measure of trainee performance, but have met with mixed success. We proposed that using an entrustability scale, where supervisors scored trainees on the level of supervision required for the case would improve the utility of compulsory mini-clinical evaluation exercise (CEX) assessments in a large anaesthesia training program. Methods. We analysed mini-CEX scores from all Australian and New Zealand College of Anaesthetists trainees submitted to an online database over a 12-month period. Supervisors' scores were adjusted for the expected supervision requirement for the case for trainees at different stages of training. We used generalisability theory to determine score reliability. Results. 7808 assessments were available for analysis. Supervision requirements decreased significantly (P < 0.05) with increased duration and level of training, supporting validity. We found moderate reliability (G > 0.7) with a feasible number of assessments. Adjusting scores against the expected supervision requirement considerably improved reliability, with G > 0.8 achieved with only nine assessments. Three per cent of trainees generated average mini-CEX scores below the expected standard. Conclusions. Using an entrustment scoring system, where supervisors score trainees on the level of supervision required, mini-CEX scores demonstrated moderate reliability within a feasible number of assessments, and evidence of validity. When scores were adjusted against an expected standard, underperforming trainees could be identified, and reliability much improved. Taken together with other evidence on trainee ability, the mini-CEX is of sufficient reliability for inclusion in high stakes decisions on trainee progression towards independent specialist practice.",
keywords = "Educational assessment, Educational measurement, Graduate, Medical education, Reliability, Workplace",
author = "Weller, {Jennifer M} and Castanelli, {D. J.} and B. Jolly and Yan Chen",
year = "2017",
month = "2",
day = "1",
doi = "10.1093/bja/aew412",
language = "English",
volume = "118",
pages = "207--214",
journal = "British Journal of Anaesthesia",
issn = "0007-0912",
publisher = "Oxford University Press",
number = "2",

}

Making robust assessments of specialist trainees' workplace performance. / Weller, Jennifer M; Castanelli, D. J.; Jolly, B.; Chen, Yan.

In: British Journal of Anaesthesia, Vol. 118, No. 2, 01.02.2017, p. 207-214.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Making robust assessments of specialist trainees' workplace performance

AU - Weller, Jennifer M

AU - Castanelli, D. J.

AU - Jolly, B.

AU - Chen, Yan

PY - 2017/2/1

Y1 - 2017/2/1

N2 - Background. Workplace-based assessments should provide a reliable measure of trainee performance, but have met with mixed success. We proposed that using an entrustability scale, where supervisors scored trainees on the level of supervision required for the case would improve the utility of compulsory mini-clinical evaluation exercise (CEX) assessments in a large anaesthesia training program. Methods. We analysed mini-CEX scores from all Australian and New Zealand College of Anaesthetists trainees submitted to an online database over a 12-month period. Supervisors' scores were adjusted for the expected supervision requirement for the case for trainees at different stages of training. We used generalisability theory to determine score reliability. Results. 7808 assessments were available for analysis. Supervision requirements decreased significantly (P < 0.05) with increased duration and level of training, supporting validity. We found moderate reliability (G > 0.7) with a feasible number of assessments. Adjusting scores against the expected supervision requirement considerably improved reliability, with G > 0.8 achieved with only nine assessments. Three per cent of trainees generated average mini-CEX scores below the expected standard. Conclusions. Using an entrustment scoring system, where supervisors score trainees on the level of supervision required, mini-CEX scores demonstrated moderate reliability within a feasible number of assessments, and evidence of validity. When scores were adjusted against an expected standard, underperforming trainees could be identified, and reliability much improved. Taken together with other evidence on trainee ability, the mini-CEX is of sufficient reliability for inclusion in high stakes decisions on trainee progression towards independent specialist practice.

AB - Background. Workplace-based assessments should provide a reliable measure of trainee performance, but have met with mixed success. We proposed that using an entrustability scale, where supervisors scored trainees on the level of supervision required for the case would improve the utility of compulsory mini-clinical evaluation exercise (CEX) assessments in a large anaesthesia training program. Methods. We analysed mini-CEX scores from all Australian and New Zealand College of Anaesthetists trainees submitted to an online database over a 12-month period. Supervisors' scores were adjusted for the expected supervision requirement for the case for trainees at different stages of training. We used generalisability theory to determine score reliability. Results. 7808 assessments were available for analysis. Supervision requirements decreased significantly (P < 0.05) with increased duration and level of training, supporting validity. We found moderate reliability (G > 0.7) with a feasible number of assessments. Adjusting scores against the expected supervision requirement considerably improved reliability, with G > 0.8 achieved with only nine assessments. Three per cent of trainees generated average mini-CEX scores below the expected standard. Conclusions. Using an entrustment scoring system, where supervisors score trainees on the level of supervision required, mini-CEX scores demonstrated moderate reliability within a feasible number of assessments, and evidence of validity. When scores were adjusted against an expected standard, underperforming trainees could be identified, and reliability much improved. Taken together with other evidence on trainee ability, the mini-CEX is of sufficient reliability for inclusion in high stakes decisions on trainee progression towards independent specialist practice.

KW - Educational assessment

KW - Educational measurement

KW - Graduate

KW - Medical education

KW - Reliability

KW - Workplace

UR - http://www.scopus.com/inward/record.url?scp=85020760129&partnerID=8YFLogxK

U2 - 10.1093/bja/aew412

DO - 10.1093/bja/aew412

M3 - Article

VL - 118

SP - 207

EP - 214

JO - British Journal of Anaesthesia

JF - British Journal of Anaesthesia

SN - 0007-0912

IS - 2

ER -