The efficacy of medical student selection tools in Australia and New Zealand

Boaz Shulruf, Warwick Bagg, Mathew Begun, Margaret Hay, Irene Lichtwark, Allison Turnock, Emma Warnecke, Timothy J Wilkinson, Phillippa J Poole

Research output: Contribution to journalArticleResearchpeer-review

2 Citations (Scopus)

Abstract

Objectives: To estimate the efficacy of selection tools employed by medical schools for predicting the binary outcomes of completing or not completing medical training and passing or failing a key examination; to investigate the potential usefulness of selection algorithms that do not allow low scores on one tool to be compensated by higher scores on other tools.

Design, setting and participants: Data from four consecutive cohorts of students (3378 students, enrolled 2007–2010) in five undergraduate medical schools in Australia and New Zealand were analysed. Predictor variables were student scores on selection tools: prior academic achievement, Undergraduate Medicine and Health Sciences Admission Test (UMAT), and selection interview. Outcome variables were graduation from the program in a timely fashion, or passing the final clinical skills assessment at the first attempt.

Main outcome measures: Optimal selection cut-scores determined by discriminant function analysis for each selection tool at each school; efficacy of different selection algorithms for predicting student outcomes.

Results: For both outcomes, the cut-scores for prior academic achievement had the greatest predictive value, with medium to very large effect sizes (0.44–1.22) at all five schools. UMAT scores and selection interviews had smaller effect sizes (0.00–0.60). Meeting one or more cut-scores was associated with a significantly greater likelihood of timely graduation in some schools but not in others.

Conclusions: An optimal cut-score can be estimated for a selection tool used for predicting an important program outcome. A “sufficient evidence” selection algorithm, founded on a non-compensatory model, is feasible, and may be useful for some schools.
Original languageEnglish
Pages (from-to)214-218
Number of pages5
JournalMedical Journal of Australia
Volume208
Issue number5
DOIs
Publication statusPublished - 19 Mar 2018

Keywords

  • medical education

Cite this

Shulruf, B., Bagg, W., Begun, M., Hay, M., Lichtwark, I., Turnock, A., ... Poole, P. J. (2018). The efficacy of medical student selection tools in Australia and New Zealand. Medical Journal of Australia, 208(5), 214-218. https://doi.org/10.5694/mja17.00400
Shulruf, Boaz ; Bagg, Warwick ; Begun, Mathew ; Hay, Margaret ; Lichtwark, Irene ; Turnock, Allison ; Warnecke, Emma ; Wilkinson, Timothy J ; Poole, Phillippa J. / The efficacy of medical student selection tools in Australia and New Zealand. In: Medical Journal of Australia. 2018 ; Vol. 208, No. 5. pp. 214-218.
@article{81fadb68c98141fb8dd8142a5adb756a,
title = "The efficacy of medical student selection tools in Australia and New Zealand",
abstract = "Objectives: To estimate the efficacy of selection tools employed by medical schools for predicting the binary outcomes of completing or not completing medical training and passing or failing a key examination; to investigate the potential usefulness of selection algorithms that do not allow low scores on one tool to be compensated by higher scores on other tools.Design, setting and participants: Data from four consecutive cohorts of students (3378 students, enrolled 2007–2010) in five undergraduate medical schools in Australia and New Zealand were analysed. Predictor variables were student scores on selection tools: prior academic achievement, Undergraduate Medicine and Health Sciences Admission Test (UMAT), and selection interview. Outcome variables were graduation from the program in a timely fashion, or passing the final clinical skills assessment at the first attempt.Main outcome measures: Optimal selection cut-scores determined by discriminant function analysis for each selection tool at each school; efficacy of different selection algorithms for predicting student outcomes.Results: For both outcomes, the cut-scores for prior academic achievement had the greatest predictive value, with medium to very large effect sizes (0.44–1.22) at all five schools. UMAT scores and selection interviews had smaller effect sizes (0.00–0.60). Meeting one or more cut-scores was associated with a significantly greater likelihood of timely graduation in some schools but not in others.Conclusions: An optimal cut-score can be estimated for a selection tool used for predicting an important program outcome. A “sufficient evidence” selection algorithm, founded on a non-compensatory model, is feasible, and may be useful for some schools.",
keywords = "medical education",
author = "Boaz Shulruf and Warwick Bagg and Mathew Begun and Margaret Hay and Irene Lichtwark and Allison Turnock and Emma Warnecke and Wilkinson, {Timothy J} and Poole, {Phillippa J}",
year = "2018",
month = "3",
day = "19",
doi = "10.5694/mja17.00400",
language = "English",
volume = "208",
pages = "214--218",
journal = "Medical Journal of Australia",
issn = "0025-729X",
publisher = "AMPCo",
number = "5",

}

Shulruf, B, Bagg, W, Begun, M, Hay, M, Lichtwark, I, Turnock, A, Warnecke, E, Wilkinson, TJ & Poole, PJ 2018, 'The efficacy of medical student selection tools in Australia and New Zealand', Medical Journal of Australia, vol. 208, no. 5, pp. 214-218. https://doi.org/10.5694/mja17.00400

The efficacy of medical student selection tools in Australia and New Zealand. / Shulruf, Boaz; Bagg, Warwick; Begun, Mathew; Hay, Margaret; Lichtwark, Irene ; Turnock, Allison; Warnecke, Emma ; Wilkinson, Timothy J; Poole, Phillippa J.

In: Medical Journal of Australia, Vol. 208, No. 5, 19.03.2018, p. 214-218.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - The efficacy of medical student selection tools in Australia and New Zealand

AU - Shulruf, Boaz

AU - Bagg, Warwick

AU - Begun, Mathew

AU - Hay, Margaret

AU - Lichtwark, Irene

AU - Turnock, Allison

AU - Warnecke, Emma

AU - Wilkinson, Timothy J

AU - Poole, Phillippa J

PY - 2018/3/19

Y1 - 2018/3/19

N2 - Objectives: To estimate the efficacy of selection tools employed by medical schools for predicting the binary outcomes of completing or not completing medical training and passing or failing a key examination; to investigate the potential usefulness of selection algorithms that do not allow low scores on one tool to be compensated by higher scores on other tools.Design, setting and participants: Data from four consecutive cohorts of students (3378 students, enrolled 2007–2010) in five undergraduate medical schools in Australia and New Zealand were analysed. Predictor variables were student scores on selection tools: prior academic achievement, Undergraduate Medicine and Health Sciences Admission Test (UMAT), and selection interview. Outcome variables were graduation from the program in a timely fashion, or passing the final clinical skills assessment at the first attempt.Main outcome measures: Optimal selection cut-scores determined by discriminant function analysis for each selection tool at each school; efficacy of different selection algorithms for predicting student outcomes.Results: For both outcomes, the cut-scores for prior academic achievement had the greatest predictive value, with medium to very large effect sizes (0.44–1.22) at all five schools. UMAT scores and selection interviews had smaller effect sizes (0.00–0.60). Meeting one or more cut-scores was associated with a significantly greater likelihood of timely graduation in some schools but not in others.Conclusions: An optimal cut-score can be estimated for a selection tool used for predicting an important program outcome. A “sufficient evidence” selection algorithm, founded on a non-compensatory model, is feasible, and may be useful for some schools.

AB - Objectives: To estimate the efficacy of selection tools employed by medical schools for predicting the binary outcomes of completing or not completing medical training and passing or failing a key examination; to investigate the potential usefulness of selection algorithms that do not allow low scores on one tool to be compensated by higher scores on other tools.Design, setting and participants: Data from four consecutive cohorts of students (3378 students, enrolled 2007–2010) in five undergraduate medical schools in Australia and New Zealand were analysed. Predictor variables were student scores on selection tools: prior academic achievement, Undergraduate Medicine and Health Sciences Admission Test (UMAT), and selection interview. Outcome variables were graduation from the program in a timely fashion, or passing the final clinical skills assessment at the first attempt.Main outcome measures: Optimal selection cut-scores determined by discriminant function analysis for each selection tool at each school; efficacy of different selection algorithms for predicting student outcomes.Results: For both outcomes, the cut-scores for prior academic achievement had the greatest predictive value, with medium to very large effect sizes (0.44–1.22) at all five schools. UMAT scores and selection interviews had smaller effect sizes (0.00–0.60). Meeting one or more cut-scores was associated with a significantly greater likelihood of timely graduation in some schools but not in others.Conclusions: An optimal cut-score can be estimated for a selection tool used for predicting an important program outcome. A “sufficient evidence” selection algorithm, founded on a non-compensatory model, is feasible, and may be useful for some schools.

KW - medical education

UR - http://www.scopus.com/inward/record.url?scp=85044379773&partnerID=8YFLogxK

U2 - 10.5694/mja17.00400

DO - 10.5694/mja17.00400

M3 - Article

VL - 208

SP - 214

EP - 218

JO - Medical Journal of Australia

JF - Medical Journal of Australia

SN - 0025-729X

IS - 5

ER -