Automatic alignment of surgical videos using kinematic data

Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, François Petitjean, Lhassane Idoumghar, Pierre Alain Muller

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

Abstract

Over the past one hundred years, the classic teaching methodology of “see one, do one, teach one” has governed the surgical education systems worldwide. With the advent of Operation Room 2.0, recording video, kinematic and many other types of data during the surgery became an easy task, thus allowing artificial intelligence systems to be deployed and used in surgical and medical practice. Recently, surgical videos has been shown to provide a structure for peer coaching enabling novice trainees to learn from experienced surgeons by replaying those videos. However, the high inter-operator variability in surgical gesture duration and execution renders learning from comparing novice to expert surgical videos a very difficult task. In this paper, we propose a novel technique to align multiple videos based on the alignment of their corresponding kinematic multivariate time series data. By leveraging the Dynamic Time Warping measure, our algorithm synchronizes a set of videos in order to show the same gesture being performed at different speed. We believe that the proposed approach is a valuable addition to the existing learning tools for surgery.

Original languageEnglish
Title of host publicationArtificial Intelligence in Medicine
Subtitle of host publication17th Conference on Artificial Intelligence in Medicine, AIME 2019 Poznan, Poland, June 26–29, 2019 Proceedings
EditorsDavid Riaño, Szymon Wilk, Annette ten Teije
Place of PublicationCham Switzerland
PublisherSpringer
Pages104-113
Number of pages10
ISBN (Electronic)9783030216429
ISBN (Print)9783030216412
DOIs
Publication statusPublished - 2019
EventArtificial Intelligence in Medicine in Europe 2019 - Poznan, Poland
Duration: 26 Jun 201929 Jun 2019
Conference number: 17th
http://aime19.aimedicine.info/

Publication series

NameLecture Notes in Computer Science
PublisherSpringer
Volume11526
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

ConferenceArtificial Intelligence in Medicine in Europe 2019
Abbreviated titleAIME 2019
CountryPoland
CityPoznan
Period26/06/1929/06/19
Internet address

Keywords

  • Dynamic Time Warping
  • Multivariate time series
  • Surgical education
  • Video synchronization

Cite this

Ismail Fawaz, H., Forestier, G., Weber, J., Petitjean, F., Idoumghar, L., & Muller, P. A. (2019). Automatic alignment of surgical videos using kinematic data. In D. Riaño, S. Wilk, & A. ten Teije (Eds.), Artificial Intelligence in Medicine: 17th Conference on Artificial Intelligence in Medicine, AIME 2019 Poznan, Poland, June 26–29, 2019 Proceedings (pp. 104-113). (Lecture Notes in Computer Science ; Vol. 11526 ). Cham Switzerland: Springer. https://doi.org/10.1007/978-3-030-21642-9_14
Ismail Fawaz, Hassan ; Forestier, Germain ; Weber, Jonathan ; Petitjean, François ; Idoumghar, Lhassane ; Muller, Pierre Alain. / Automatic alignment of surgical videos using kinematic data. Artificial Intelligence in Medicine: 17th Conference on Artificial Intelligence in Medicine, AIME 2019 Poznan, Poland, June 26–29, 2019 Proceedings. editor / David Riaño ; Szymon Wilk ; Annette ten Teije. Cham Switzerland : Springer, 2019. pp. 104-113 (Lecture Notes in Computer Science ).
@inproceedings{53dd79a0a8d540cfac3189aeb91927ad,
title = "Automatic alignment of surgical videos using kinematic data",
abstract = "Over the past one hundred years, the classic teaching methodology of “see one, do one, teach one” has governed the surgical education systems worldwide. With the advent of Operation Room 2.0, recording video, kinematic and many other types of data during the surgery became an easy task, thus allowing artificial intelligence systems to be deployed and used in surgical and medical practice. Recently, surgical videos has been shown to provide a structure for peer coaching enabling novice trainees to learn from experienced surgeons by replaying those videos. However, the high inter-operator variability in surgical gesture duration and execution renders learning from comparing novice to expert surgical videos a very difficult task. In this paper, we propose a novel technique to align multiple videos based on the alignment of their corresponding kinematic multivariate time series data. By leveraging the Dynamic Time Warping measure, our algorithm synchronizes a set of videos in order to show the same gesture being performed at different speed. We believe that the proposed approach is a valuable addition to the existing learning tools for surgery.",
keywords = "Dynamic Time Warping, Multivariate time series, Surgical education, Video synchronization",
author = "{Ismail Fawaz}, Hassan and Germain Forestier and Jonathan Weber and Fran{\cc}ois Petitjean and Lhassane Idoumghar and Muller, {Pierre Alain}",
year = "2019",
doi = "10.1007/978-3-030-21642-9_14",
language = "English",
isbn = "9783030216412",
series = "Lecture Notes in Computer Science",
publisher = "Springer",
pages = "104--113",
editor = "David Ria{\~n}o and Szymon Wilk and {ten Teije}, Annette",
booktitle = "Artificial Intelligence in Medicine",

}

Ismail Fawaz, H, Forestier, G, Weber, J, Petitjean, F, Idoumghar, L & Muller, PA 2019, Automatic alignment of surgical videos using kinematic data. in D Riaño, S Wilk & A ten Teije (eds), Artificial Intelligence in Medicine: 17th Conference on Artificial Intelligence in Medicine, AIME 2019 Poznan, Poland, June 26–29, 2019 Proceedings. Lecture Notes in Computer Science , vol. 11526 , Springer, Cham Switzerland, pp. 104-113, Artificial Intelligence in Medicine in Europe 2019, Poznan, Poland, 26/06/19. https://doi.org/10.1007/978-3-030-21642-9_14

Automatic alignment of surgical videos using kinematic data. / Ismail Fawaz, Hassan; Forestier, Germain; Weber, Jonathan; Petitjean, François; Idoumghar, Lhassane; Muller, Pierre Alain.

Artificial Intelligence in Medicine: 17th Conference on Artificial Intelligence in Medicine, AIME 2019 Poznan, Poland, June 26–29, 2019 Proceedings. ed. / David Riaño; Szymon Wilk; Annette ten Teije. Cham Switzerland : Springer, 2019. p. 104-113 (Lecture Notes in Computer Science ; Vol. 11526 ).

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

TY - GEN

T1 - Automatic alignment of surgical videos using kinematic data

AU - Ismail Fawaz, Hassan

AU - Forestier, Germain

AU - Weber, Jonathan

AU - Petitjean, François

AU - Idoumghar, Lhassane

AU - Muller, Pierre Alain

PY - 2019

Y1 - 2019

N2 - Over the past one hundred years, the classic teaching methodology of “see one, do one, teach one” has governed the surgical education systems worldwide. With the advent of Operation Room 2.0, recording video, kinematic and many other types of data during the surgery became an easy task, thus allowing artificial intelligence systems to be deployed and used in surgical and medical practice. Recently, surgical videos has been shown to provide a structure for peer coaching enabling novice trainees to learn from experienced surgeons by replaying those videos. However, the high inter-operator variability in surgical gesture duration and execution renders learning from comparing novice to expert surgical videos a very difficult task. In this paper, we propose a novel technique to align multiple videos based on the alignment of their corresponding kinematic multivariate time series data. By leveraging the Dynamic Time Warping measure, our algorithm synchronizes a set of videos in order to show the same gesture being performed at different speed. We believe that the proposed approach is a valuable addition to the existing learning tools for surgery.

AB - Over the past one hundred years, the classic teaching methodology of “see one, do one, teach one” has governed the surgical education systems worldwide. With the advent of Operation Room 2.0, recording video, kinematic and many other types of data during the surgery became an easy task, thus allowing artificial intelligence systems to be deployed and used in surgical and medical practice. Recently, surgical videos has been shown to provide a structure for peer coaching enabling novice trainees to learn from experienced surgeons by replaying those videos. However, the high inter-operator variability in surgical gesture duration and execution renders learning from comparing novice to expert surgical videos a very difficult task. In this paper, we propose a novel technique to align multiple videos based on the alignment of their corresponding kinematic multivariate time series data. By leveraging the Dynamic Time Warping measure, our algorithm synchronizes a set of videos in order to show the same gesture being performed at different speed. We believe that the proposed approach is a valuable addition to the existing learning tools for surgery.

KW - Dynamic Time Warping

KW - Multivariate time series

KW - Surgical education

KW - Video synchronization

UR - http://www.scopus.com/inward/record.url?scp=85068325501&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-21642-9_14

DO - 10.1007/978-3-030-21642-9_14

M3 - Conference Paper

SN - 9783030216412

T3 - Lecture Notes in Computer Science

SP - 104

EP - 113

BT - Artificial Intelligence in Medicine

A2 - Riaño, David

A2 - Wilk, Szymon

A2 - ten Teije, Annette

PB - Springer

CY - Cham Switzerland

ER -

Ismail Fawaz H, Forestier G, Weber J, Petitjean F, Idoumghar L, Muller PA. Automatic alignment of surgical videos using kinematic data. In Riaño D, Wilk S, ten Teije A, editors, Artificial Intelligence in Medicine: 17th Conference on Artificial Intelligence in Medicine, AIME 2019 Poznan, Poland, June 26–29, 2019 Proceedings. Cham Switzerland: Springer. 2019. p. 104-113. (Lecture Notes in Computer Science ). https://doi.org/10.1007/978-3-030-21642-9_14