Validation of ‘Somnivore’, a Machine Learning Algorithm for Automated Scoring and Analysis of Polysomnography Data

Giancarlo Allocca, Sherie Ma, Davide Martelli, Matteo Cerri, Flavia Del Vecchio, Stefano Bastianini, Giovanna Zoccoli, Roberto Amici, Stephen R. Morairty, Anne E. Aulsebrook, Shaun Blackburn, John A. Lesku, Niels C. Rattenborg, Alexei L. Vyssotski, Emma Wams, Kate Porcheret, Katharina Wulff, Russell Foster, Julia K.M. Chan, Christian L. Nicholas & 3 others Dean R. Freestone, Leigh A. Johnston, Andrew L. Gundlach

Research output: Contribution to journalArticleResearchpeer-review

2 Citations (Scopus)

Abstract

Manual scoring of polysomnography data is labor-intensive and time-consuming, and most existing software does not account for subjective differences and user variability. Therefore, we evaluated a supervised machine learning algorithm, SomnivoreTM, for automated wake–sleep stage classification. We designed an algorithm that extracts features from various input channels, following a brief session of manual scoring, and provides automated wake-sleep stage classification for each recording. For algorithm validation, polysomnography data was obtained from independent laboratories, and include normal, cognitively-impaired, and alcohol-treated human subjects (total n = 52), narcoleptic mice and drug-treated rats (total n = 56), and pigeons (n = 5). Training and testing sets for validation were previously scored manually by 1–2 trained sleep technologists from each laboratory. F-measure was used to assess precision and sensitivity for statistical analysis of classifier output and human scorer agreement. The algorithm gave high concordance with manual visual scoring across all human data (wake 0.91 ± 0.01; N1 0.57 ± 0.01; N2 0.81 ± 0.01; N3 0.86 ± 0.01; REM 0.87 ± 0.01), which was comparable to manual inter-scorer agreement on all stages. Similarly, high concordance was observed across all rodent (wake 0.95 ± 0.01; NREM 0.94 ± 0.01; REM 0.91 ± 0.01) and pigeon (wake 0.96 ± 0.006; NREM 0.97 ± 0.01; REM 0.86 ± 0.02) data. Effects of classifier learning from single signal inputs, simple stage reclassification, automated removal of transition epochs, and training set size were also examined. In summary, we have developed a polysomnography analysis program for automated sleep-stage classification of data from diverse species. Somnivore enables flexible, accurate, and high-throughput analysis of experimental and clinical sleep studies.

Original languageEnglish
Article number207
Pages (from-to)1-18
Number of pages18
JournalFrontiers in Neuroscience
Volume13
DOIs
Publication statusPublished - 2019
Externally publishedYes

Keywords

  • Machine learning algorithms
  • Polysomnography
  • Signal processing algorithms
  • Sleep stage classification
  • Sleep stage scoring
  • Wake

Cite this

Allocca, Giancarlo ; Ma, Sherie ; Martelli, Davide ; Cerri, Matteo ; Del Vecchio, Flavia ; Bastianini, Stefano ; Zoccoli, Giovanna ; Amici, Roberto ; Morairty, Stephen R. ; Aulsebrook, Anne E. ; Blackburn, Shaun ; Lesku, John A. ; Rattenborg, Niels C. ; Vyssotski, Alexei L. ; Wams, Emma ; Porcheret, Kate ; Wulff, Katharina ; Foster, Russell ; Chan, Julia K.M. ; Nicholas, Christian L. ; Freestone, Dean R. ; Johnston, Leigh A. ; Gundlach, Andrew L. / Validation of ‘Somnivore’, a Machine Learning Algorithm for Automated Scoring and Analysis of Polysomnography Data. In: Frontiers in Neuroscience. 2019 ; Vol. 13. pp. 1-18.
@article{29f896168d924adfbd81ad030ed2f34f,
title = "Validation of ‘Somnivore’, a Machine Learning Algorithm for Automated Scoring and Analysis of Polysomnography Data",
abstract = "Manual scoring of polysomnography data is labor-intensive and time-consuming, and most existing software does not account for subjective differences and user variability. Therefore, we evaluated a supervised machine learning algorithm, SomnivoreTM, for automated wake–sleep stage classification. We designed an algorithm that extracts features from various input channels, following a brief session of manual scoring, and provides automated wake-sleep stage classification for each recording. For algorithm validation, polysomnography data was obtained from independent laboratories, and include normal, cognitively-impaired, and alcohol-treated human subjects (total n = 52), narcoleptic mice and drug-treated rats (total n = 56), and pigeons (n = 5). Training and testing sets for validation were previously scored manually by 1–2 trained sleep technologists from each laboratory. F-measure was used to assess precision and sensitivity for statistical analysis of classifier output and human scorer agreement. The algorithm gave high concordance with manual visual scoring across all human data (wake 0.91 ± 0.01; N1 0.57 ± 0.01; N2 0.81 ± 0.01; N3 0.86 ± 0.01; REM 0.87 ± 0.01), which was comparable to manual inter-scorer agreement on all stages. Similarly, high concordance was observed across all rodent (wake 0.95 ± 0.01; NREM 0.94 ± 0.01; REM 0.91 ± 0.01) and pigeon (wake 0.96 ± 0.006; NREM 0.97 ± 0.01; REM 0.86 ± 0.02) data. Effects of classifier learning from single signal inputs, simple stage reclassification, automated removal of transition epochs, and training set size were also examined. In summary, we have developed a polysomnography analysis program for automated sleep-stage classification of data from diverse species. Somnivore enables flexible, accurate, and high-throughput analysis of experimental and clinical sleep studies.",
keywords = "Machine learning algorithms, Polysomnography, Signal processing algorithms, Sleep stage classification, Sleep stage scoring, Wake",
author = "Giancarlo Allocca and Sherie Ma and Davide Martelli and Matteo Cerri and {Del Vecchio}, Flavia and Stefano Bastianini and Giovanna Zoccoli and Roberto Amici and Morairty, {Stephen R.} and Aulsebrook, {Anne E.} and Shaun Blackburn and Lesku, {John A.} and Rattenborg, {Niels C.} and Vyssotski, {Alexei L.} and Emma Wams and Kate Porcheret and Katharina Wulff and Russell Foster and Chan, {Julia K.M.} and Nicholas, {Christian L.} and Freestone, {Dean R.} and Johnston, {Leigh A.} and Gundlach, {Andrew L.}",
year = "2019",
doi = "10.3389/fnins.2019.00207",
language = "English",
volume = "13",
pages = "1--18",
journal = "Frontiers in Neuroscience",
issn = "1662-453X",
publisher = "Frontiers Research Foundation",

}

Allocca, G, Ma, S, Martelli, D, Cerri, M, Del Vecchio, F, Bastianini, S, Zoccoli, G, Amici, R, Morairty, SR, Aulsebrook, AE, Blackburn, S, Lesku, JA, Rattenborg, NC, Vyssotski, AL, Wams, E, Porcheret, K, Wulff, K, Foster, R, Chan, JKM, Nicholas, CL, Freestone, DR, Johnston, LA & Gundlach, AL 2019, 'Validation of ‘Somnivore’, a Machine Learning Algorithm for Automated Scoring and Analysis of Polysomnography Data', Frontiers in Neuroscience, vol. 13, 207, pp. 1-18. https://doi.org/10.3389/fnins.2019.00207

Validation of ‘Somnivore’, a Machine Learning Algorithm for Automated Scoring and Analysis of Polysomnography Data. / Allocca, Giancarlo; Ma, Sherie; Martelli, Davide; Cerri, Matteo; Del Vecchio, Flavia; Bastianini, Stefano; Zoccoli, Giovanna; Amici, Roberto; Morairty, Stephen R.; Aulsebrook, Anne E.; Blackburn, Shaun; Lesku, John A.; Rattenborg, Niels C.; Vyssotski, Alexei L.; Wams, Emma; Porcheret, Kate; Wulff, Katharina; Foster, Russell; Chan, Julia K.M.; Nicholas, Christian L.; Freestone, Dean R.; Johnston, Leigh A.; Gundlach, Andrew L.

In: Frontiers in Neuroscience, Vol. 13, 207, 2019, p. 1-18.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Validation of ‘Somnivore’, a Machine Learning Algorithm for Automated Scoring and Analysis of Polysomnography Data

AU - Allocca, Giancarlo

AU - Ma, Sherie

AU - Martelli, Davide

AU - Cerri, Matteo

AU - Del Vecchio, Flavia

AU - Bastianini, Stefano

AU - Zoccoli, Giovanna

AU - Amici, Roberto

AU - Morairty, Stephen R.

AU - Aulsebrook, Anne E.

AU - Blackburn, Shaun

AU - Lesku, John A.

AU - Rattenborg, Niels C.

AU - Vyssotski, Alexei L.

AU - Wams, Emma

AU - Porcheret, Kate

AU - Wulff, Katharina

AU - Foster, Russell

AU - Chan, Julia K.M.

AU - Nicholas, Christian L.

AU - Freestone, Dean R.

AU - Johnston, Leigh A.

AU - Gundlach, Andrew L.

PY - 2019

Y1 - 2019

N2 - Manual scoring of polysomnography data is labor-intensive and time-consuming, and most existing software does not account for subjective differences and user variability. Therefore, we evaluated a supervised machine learning algorithm, SomnivoreTM, for automated wake–sleep stage classification. We designed an algorithm that extracts features from various input channels, following a brief session of manual scoring, and provides automated wake-sleep stage classification for each recording. For algorithm validation, polysomnography data was obtained from independent laboratories, and include normal, cognitively-impaired, and alcohol-treated human subjects (total n = 52), narcoleptic mice and drug-treated rats (total n = 56), and pigeons (n = 5). Training and testing sets for validation were previously scored manually by 1–2 trained sleep technologists from each laboratory. F-measure was used to assess precision and sensitivity for statistical analysis of classifier output and human scorer agreement. The algorithm gave high concordance with manual visual scoring across all human data (wake 0.91 ± 0.01; N1 0.57 ± 0.01; N2 0.81 ± 0.01; N3 0.86 ± 0.01; REM 0.87 ± 0.01), which was comparable to manual inter-scorer agreement on all stages. Similarly, high concordance was observed across all rodent (wake 0.95 ± 0.01; NREM 0.94 ± 0.01; REM 0.91 ± 0.01) and pigeon (wake 0.96 ± 0.006; NREM 0.97 ± 0.01; REM 0.86 ± 0.02) data. Effects of classifier learning from single signal inputs, simple stage reclassification, automated removal of transition epochs, and training set size were also examined. In summary, we have developed a polysomnography analysis program for automated sleep-stage classification of data from diverse species. Somnivore enables flexible, accurate, and high-throughput analysis of experimental and clinical sleep studies.

AB - Manual scoring of polysomnography data is labor-intensive and time-consuming, and most existing software does not account for subjective differences and user variability. Therefore, we evaluated a supervised machine learning algorithm, SomnivoreTM, for automated wake–sleep stage classification. We designed an algorithm that extracts features from various input channels, following a brief session of manual scoring, and provides automated wake-sleep stage classification for each recording. For algorithm validation, polysomnography data was obtained from independent laboratories, and include normal, cognitively-impaired, and alcohol-treated human subjects (total n = 52), narcoleptic mice and drug-treated rats (total n = 56), and pigeons (n = 5). Training and testing sets for validation were previously scored manually by 1–2 trained sleep technologists from each laboratory. F-measure was used to assess precision and sensitivity for statistical analysis of classifier output and human scorer agreement. The algorithm gave high concordance with manual visual scoring across all human data (wake 0.91 ± 0.01; N1 0.57 ± 0.01; N2 0.81 ± 0.01; N3 0.86 ± 0.01; REM 0.87 ± 0.01), which was comparable to manual inter-scorer agreement on all stages. Similarly, high concordance was observed across all rodent (wake 0.95 ± 0.01; NREM 0.94 ± 0.01; REM 0.91 ± 0.01) and pigeon (wake 0.96 ± 0.006; NREM 0.97 ± 0.01; REM 0.86 ± 0.02) data. Effects of classifier learning from single signal inputs, simple stage reclassification, automated removal of transition epochs, and training set size were also examined. In summary, we have developed a polysomnography analysis program for automated sleep-stage classification of data from diverse species. Somnivore enables flexible, accurate, and high-throughput analysis of experimental and clinical sleep studies.

KW - Machine learning algorithms

KW - Polysomnography

KW - Signal processing algorithms

KW - Sleep stage classification

KW - Sleep stage scoring

KW - Wake

UR - http://www.scopus.com/inward/record.url?scp=85065910116&partnerID=8YFLogxK

U2 - 10.3389/fnins.2019.00207

DO - 10.3389/fnins.2019.00207

M3 - Article

VL - 13

SP - 1

EP - 18

JO - Frontiers in Neuroscience

JF - Frontiers in Neuroscience

SN - 1662-453X

M1 - 207

ER -