An extended protocol for usability validation of medical devices: Research design and reference model

Martin Schmettow, Raphaela Schnittker, Jan Maarten Schraagen

Research output: Contribution to journalArticleResearchpeer-review

Abstract

This paper proposes and demonstrates an extended protocol for usability validation testing of medical devices. A review of currently used methods for the usability evaluation of medical devices revealed two main shortcomings. Firstly, the lack of methods to closely trace the interaction sequences and derive performance measures. Secondly, a prevailing focus on cross-sectional validation studies, ignoring the issues of learnability and training. The U.S. Federal Drug and Food Administration's recent proposal for a validation testing protocol for medical devices is then extended to address these shortcomings: (1) a novel process measure ‘normative path deviations’ is introduced that is useful for both quantitative and qualitative usability studies and (2) a longitudinal, completely within-subject study design is presented that assesses learnability, training effects and allows analysis of diversity of users. A reference regression model is introduced to analyze data from this and similar studies, drawing upon generalized linear mixed-effects models and a Bayesian estimation approach. The extended protocol is implemented and demonstrated in a study comparing a novel syringe infusion pump prototype to an existing design with a sample of 25 healthcare professionals. Strong performance differences between designs were observed with a variety of usability measures, as well as varying training-on-the-job effects. We discuss our findings with regard to validation testing guidelines, reflect on the extensions and discuss the perspectives they add to the validation process.

Original languageEnglish
Pages (from-to)99-114
Number of pages16
JournalJournal of Biomedical Informatics
Volume69
DOIs
Publication statusPublished - 1 May 2017

Keywords

  • Generalized linear mixed-effects models
  • Human Factors
  • Longitudinal research
  • Medical device design
  • Patient safety
  • Usability testing

Cite this

@article{792f56d9fcce48baa20f13e934549d6f,
title = "An extended protocol for usability validation of medical devices: Research design and reference model",
abstract = "This paper proposes and demonstrates an extended protocol for usability validation testing of medical devices. A review of currently used methods for the usability evaluation of medical devices revealed two main shortcomings. Firstly, the lack of methods to closely trace the interaction sequences and derive performance measures. Secondly, a prevailing focus on cross-sectional validation studies, ignoring the issues of learnability and training. The U.S. Federal Drug and Food Administration's recent proposal for a validation testing protocol for medical devices is then extended to address these shortcomings: (1) a novel process measure ‘normative path deviations’ is introduced that is useful for both quantitative and qualitative usability studies and (2) a longitudinal, completely within-subject study design is presented that assesses learnability, training effects and allows analysis of diversity of users. A reference regression model is introduced to analyze data from this and similar studies, drawing upon generalized linear mixed-effects models and a Bayesian estimation approach. The extended protocol is implemented and demonstrated in a study comparing a novel syringe infusion pump prototype to an existing design with a sample of 25 healthcare professionals. Strong performance differences between designs were observed with a variety of usability measures, as well as varying training-on-the-job effects. We discuss our findings with regard to validation testing guidelines, reflect on the extensions and discuss the perspectives they add to the validation process.",
keywords = "Generalized linear mixed-effects models, Human Factors, Longitudinal research, Medical device design, Patient safety, Usability testing",
author = "Martin Schmettow and Raphaela Schnittker and Schraagen, {Jan Maarten}",
year = "2017",
month = "5",
day = "1",
doi = "10.1016/j.jbi.2017.03.010",
language = "English",
volume = "69",
pages = "99--114",
journal = "Journal of Biomedical Informatics",
issn = "1532-0464",
publisher = "Elsevier",

}

An extended protocol for usability validation of medical devices : Research design and reference model. / Schmettow, Martin; Schnittker, Raphaela; Schraagen, Jan Maarten.

In: Journal of Biomedical Informatics, Vol. 69, 01.05.2017, p. 99-114.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - An extended protocol for usability validation of medical devices

T2 - Research design and reference model

AU - Schmettow, Martin

AU - Schnittker, Raphaela

AU - Schraagen, Jan Maarten

PY - 2017/5/1

Y1 - 2017/5/1

N2 - This paper proposes and demonstrates an extended protocol for usability validation testing of medical devices. A review of currently used methods for the usability evaluation of medical devices revealed two main shortcomings. Firstly, the lack of methods to closely trace the interaction sequences and derive performance measures. Secondly, a prevailing focus on cross-sectional validation studies, ignoring the issues of learnability and training. The U.S. Federal Drug and Food Administration's recent proposal for a validation testing protocol for medical devices is then extended to address these shortcomings: (1) a novel process measure ‘normative path deviations’ is introduced that is useful for both quantitative and qualitative usability studies and (2) a longitudinal, completely within-subject study design is presented that assesses learnability, training effects and allows analysis of diversity of users. A reference regression model is introduced to analyze data from this and similar studies, drawing upon generalized linear mixed-effects models and a Bayesian estimation approach. The extended protocol is implemented and demonstrated in a study comparing a novel syringe infusion pump prototype to an existing design with a sample of 25 healthcare professionals. Strong performance differences between designs were observed with a variety of usability measures, as well as varying training-on-the-job effects. We discuss our findings with regard to validation testing guidelines, reflect on the extensions and discuss the perspectives they add to the validation process.

AB - This paper proposes and demonstrates an extended protocol for usability validation testing of medical devices. A review of currently used methods for the usability evaluation of medical devices revealed two main shortcomings. Firstly, the lack of methods to closely trace the interaction sequences and derive performance measures. Secondly, a prevailing focus on cross-sectional validation studies, ignoring the issues of learnability and training. The U.S. Federal Drug and Food Administration's recent proposal for a validation testing protocol for medical devices is then extended to address these shortcomings: (1) a novel process measure ‘normative path deviations’ is introduced that is useful for both quantitative and qualitative usability studies and (2) a longitudinal, completely within-subject study design is presented that assesses learnability, training effects and allows analysis of diversity of users. A reference regression model is introduced to analyze data from this and similar studies, drawing upon generalized linear mixed-effects models and a Bayesian estimation approach. The extended protocol is implemented and demonstrated in a study comparing a novel syringe infusion pump prototype to an existing design with a sample of 25 healthcare professionals. Strong performance differences between designs were observed with a variety of usability measures, as well as varying training-on-the-job effects. We discuss our findings with regard to validation testing guidelines, reflect on the extensions and discuss the perspectives they add to the validation process.

KW - Generalized linear mixed-effects models

KW - Human Factors

KW - Longitudinal research

KW - Medical device design

KW - Patient safety

KW - Usability testing

UR - http://www.scopus.com/inward/record.url?scp=85017200479&partnerID=8YFLogxK

U2 - 10.1016/j.jbi.2017.03.010

DO - 10.1016/j.jbi.2017.03.010

M3 - Article

VL - 69

SP - 99

EP - 114

JO - Journal of Biomedical Informatics

JF - Journal of Biomedical Informatics

SN - 1532-0464

ER -