Interassessor agreement of portfolio-based competency assessment for orthotists/prosthetists in Australia: a mixed method study

Michael P. Dillon, Louise Puli, Emily Ridgewell, Sarah P. Anderson, Neville Chiavaroli, Leigh Clarke

Research output: Contribution to journalArticleResearchpeer-review

Abstract

Background:Internationally qualified orthotists/prosthetists who want to practice in Australia must pass a portfolio-based competency assessment. Testing the agreement between independent assessors is important to engender confidence in the assessment, and continually improve the processes.Objectives:To quantify interassessor agreement for all 68 performance indicators in the Australian Orthotic Prosthetic Association's Entry Level Competency Standards and where there was significant disagreement between assessors, to explore the reasons why.Study design:Mixed methods: explanatory sequential.Method:Fifteen portfolios were assigned to independent assessors. Assessors determined whether the evidence presented met the requirements of each performance indicator. Interassessor agreement was calculated using Gwet's Agreement Coefficient 1 (AC1), and these data informed semistructured interviews to explore the reasons for disagreement.Results:Most performance indicators (87%) had moderate to substantial agreement (AC1 > 0.71), which could be attributed to a variety of factors including the use of a simple assessment rubric with supporting guidelines and assessor training to establish shared expectations. The remaining performance indicators (13%) had fair to slight agreement (AC1 ≤ 0.7). Interviews with assessors suggested that disagreement could be attributed to the complexity of some performance indicators, unconscious bias, and the appropriateness of the evidence presented.Conclusions:Although most performance indicators in Australian Orthotic Prosthetic Association's Entry Level Competency Standard were associated with moderate to substantial interassessor agreement, there are opportunities to improve agreement by simplifying the wording of some performance indicators and revising guidelines to help applicants curate the most appropriate evidence for each performance indicator.

Original languageEnglish
Pages (from-to)276-288
Number of pages13
JournalProsthetics and Orthotics International
Volume45
Issue number3
DOIs
Publication statusPublished - Jun 2021
Externally publishedYes

Keywords

  • agreement
  • competency
  • orthotist
  • portfolio
  • professional regulation
  • prosthetist
  • reliability
  • standards

Cite this