An explainable deep fusion network for affect recognition using physiological signals

Jionghao Lin, Shirui Pan, Cheng Siong Lee, Sharon Oviatt

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

71 Citations (Scopus)

Abstract

Affective computing is an emerging research area which provides insights on human's mental state through human-machine interaction. During the interaction process, bio-signal analysis is essential to detect human affective changes. Currently, machine learning methods to analyse bio-signals are the state of the art to detect the affective states, but most empirical works mainly deploy traditional machine learning methods rather than deep learning models due to the need for explainability. In this paper, we propose a deep learning model to process multimodal-multisensory bio-signals for affect recognition. It supports batch training for different sampling rate signals at the same time, and our results show significant improvement compared to the state of the art. Furthermore, the results are interpreted at the sensor- and signal- level to improve the explainaibility of our deep learning model.

Original languageEnglish
Title of host publicationProceedings of the 28th ACM International Conference on Information and Knowledge Management
EditorsPeng Cui, Elke Rundensteiner, David Carmel, Qi He, Jeffrey Xu Yu
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Pages2069-2072
Number of pages4
ISBN (Electronic)9781450369763
DOIs
Publication statusPublished - 2019
EventACM International Conference on Information and Knowledge Management 2019 - Beijing, China
Duration: 3 Nov 20197 Nov 2019
Conference number: 28th
http://www.cikm2019.net/
https://dl.acm.org/doi/proceedings/10.1145/3357384

Conference

ConferenceACM International Conference on Information and Knowledge Management 2019
Abbreviated titleCIKM 2019
Country/TerritoryChina
CityBeijing
Period3/11/197/11/19
Internet address

Keywords

  • Affect recognition
  • Deep learning
  • Explainability
  • Multimodal fusion

Cite this