An explainable deep fusion network for affect recognition using physiological signals

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

Abstract

Affective computing is an emerging research area which provides insights on human's mental state through human-machine interaction. During the interaction process, bio-signal analysis is essential to detect human affective changes. Currently, machine learning methods to analyse bio-signals are the state of the art to detect the affective states, but most empirical works mainly deploy traditional machine learning methods rather than deep learning models due to the need for explainability. In this paper, we propose a deep learning model to process multimodal-multisensory bio-signals for affect recognition. It supports batch training for different sampling rate signals at the same time, and our results show significant improvement compared to the state of the art. Furthermore, the results are interpreted at the sensor- and signal- level to improve the explainaibility of our deep learning model.

Original languageEnglish
Title of host publicationProceedings of the 28th ACM International Conference on Information and Knowledge Management
EditorsPeng Cui, Elke Rundensteiner, David Carmel, Qi He, Jeffrey Xu Yu
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Pages2069-2072
Number of pages4
ISBN (Electronic)9781450369763
DOIs
Publication statusPublished - 2019
EventACM International Conference on Information and Knowledge Management 2019 - Beijing, China
Duration: 3 Nov 20197 Nov 2019
Conference number: 28th
http://www.cikm2019.net/

Conference

ConferenceACM International Conference on Information and Knowledge Management 2019
Abbreviated titleCIKM 2019
CountryChina
CityBeijing
Period3/11/197/11/19
Internet address

Keywords

  • Affect recognition
  • Deep learning
  • Explainability
  • Multimodal fusion

Cite this

Lin, J., Pan, S., Lee, C. S., & Oviatt, S. (2019). An explainable deep fusion network for affect recognition using physiological signals. In P. Cui, E. Rundensteiner, D. Carmel, Q. He, & J. Xu Yu (Eds.), Proceedings of the 28th ACM International Conference on Information and Knowledge Management (pp. 2069-2072). New York NY USA: Association for Computing Machinery (ACM). https://doi.org/10.1145/3357384.3358160
Lin, Jionghao ; Pan, Shirui ; Lee, Cheng Siong ; Oviatt, Sharon. / An explainable deep fusion network for affect recognition using physiological signals. Proceedings of the 28th ACM International Conference on Information and Knowledge Management. editor / Peng Cui ; Elke Rundensteiner ; David Carmel ; Qi He ; Jeffrey Xu Yu. New York NY USA : Association for Computing Machinery (ACM), 2019. pp. 2069-2072
@inproceedings{1c59fad99a4243b48e5cb1fd5bce92aa,
title = "An explainable deep fusion network for affect recognition using physiological signals",
abstract = "Affective computing is an emerging research area which provides insights on human's mental state through human-machine interaction. During the interaction process, bio-signal analysis is essential to detect human affective changes. Currently, machine learning methods to analyse bio-signals are the state of the art to detect the affective states, but most empirical works mainly deploy traditional machine learning methods rather than deep learning models due to the need for explainability. In this paper, we propose a deep learning model to process multimodal-multisensory bio-signals for affect recognition. It supports batch training for different sampling rate signals at the same time, and our results show significant improvement compared to the state of the art. Furthermore, the results are interpreted at the sensor- and signal- level to improve the explainaibility of our deep learning model.",
keywords = "Affect recognition, Deep learning, Explainability, Multimodal fusion",
author = "Jionghao Lin and Shirui Pan and Lee, {Cheng Siong} and Sharon Oviatt",
year = "2019",
doi = "10.1145/3357384.3358160",
language = "English",
pages = "2069--2072",
editor = "Cui, {Peng } and Rundensteiner, {Elke } and Carmel, {David } and He, {Qi } and {Xu Yu}, {Jeffrey }",
booktitle = "Proceedings of the 28th ACM International Conference on Information and Knowledge Management",
publisher = "Association for Computing Machinery (ACM)",
address = "United States of America",

}

Lin, J, Pan, S, Lee, CS & Oviatt, S 2019, An explainable deep fusion network for affect recognition using physiological signals. in P Cui, E Rundensteiner, D Carmel, Q He & J Xu Yu (eds), Proceedings of the 28th ACM International Conference on Information and Knowledge Management. Association for Computing Machinery (ACM), New York NY USA, pp. 2069-2072, ACM International Conference on Information and Knowledge Management 2019, Beijing, China, 3/11/19. https://doi.org/10.1145/3357384.3358160

An explainable deep fusion network for affect recognition using physiological signals. / Lin, Jionghao; Pan, Shirui; Lee, Cheng Siong; Oviatt, Sharon.

Proceedings of the 28th ACM International Conference on Information and Knowledge Management. ed. / Peng Cui; Elke Rundensteiner; David Carmel; Qi He; Jeffrey Xu Yu. New York NY USA : Association for Computing Machinery (ACM), 2019. p. 2069-2072.

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

TY - GEN

T1 - An explainable deep fusion network for affect recognition using physiological signals

AU - Lin, Jionghao

AU - Pan, Shirui

AU - Lee, Cheng Siong

AU - Oviatt, Sharon

PY - 2019

Y1 - 2019

N2 - Affective computing is an emerging research area which provides insights on human's mental state through human-machine interaction. During the interaction process, bio-signal analysis is essential to detect human affective changes. Currently, machine learning methods to analyse bio-signals are the state of the art to detect the affective states, but most empirical works mainly deploy traditional machine learning methods rather than deep learning models due to the need for explainability. In this paper, we propose a deep learning model to process multimodal-multisensory bio-signals for affect recognition. It supports batch training for different sampling rate signals at the same time, and our results show significant improvement compared to the state of the art. Furthermore, the results are interpreted at the sensor- and signal- level to improve the explainaibility of our deep learning model.

AB - Affective computing is an emerging research area which provides insights on human's mental state through human-machine interaction. During the interaction process, bio-signal analysis is essential to detect human affective changes. Currently, machine learning methods to analyse bio-signals are the state of the art to detect the affective states, but most empirical works mainly deploy traditional machine learning methods rather than deep learning models due to the need for explainability. In this paper, we propose a deep learning model to process multimodal-multisensory bio-signals for affect recognition. It supports batch training for different sampling rate signals at the same time, and our results show significant improvement compared to the state of the art. Furthermore, the results are interpreted at the sensor- and signal- level to improve the explainaibility of our deep learning model.

KW - Affect recognition

KW - Deep learning

KW - Explainability

KW - Multimodal fusion

UR - http://www.scopus.com/inward/record.url?scp=85075478890&partnerID=8YFLogxK

U2 - 10.1145/3357384.3358160

DO - 10.1145/3357384.3358160

M3 - Conference Paper

SP - 2069

EP - 2072

BT - Proceedings of the 28th ACM International Conference on Information and Knowledge Management

A2 - Cui, Peng

A2 - Rundensteiner, Elke

A2 - Carmel, David

A2 - He, Qi

A2 - Xu Yu, Jeffrey

PB - Association for Computing Machinery (ACM)

CY - New York NY USA

ER -

Lin J, Pan S, Lee CS, Oviatt S. An explainable deep fusion network for affect recognition using physiological signals. In Cui P, Rundensteiner E, Carmel D, He Q, Xu Yu J, editors, Proceedings of the 28th ACM International Conference on Information and Knowledge Management. New York NY USA: Association for Computing Machinery (ACM). 2019. p. 2069-2072 https://doi.org/10.1145/3357384.3358160