Audio-visual weakly supervised approach for apathy detection in the elderly

Garima Sharma, Jyoti Joshi, Radia Zeghari, Rachid Guerchouche

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

1 Citation (Scopus)


Apathy is manifested as lack of feelings or emotions in several neurological and psychological disorders. Hence, directly impairing the display of emotion through facial expressions and speech. Current practices of prediction of apathy heavily rely on clinical diagnosis, an expert interviewing a patient or reports from patients' family members. The dependence on an expert and the human bias in its examination results in under-diagnosis of condition. In this paper, a multimodal multi-instance learning based method is proposed for automatic apathy detection. There are several challenges present while automating the process. Some of which are - recognizing emotions in elderly people, correct identification of emotions in a conversation and identifying differences between emotional responses from apathetic and non apathetic cohorts. The proposed method uses the audio and visual information in a weakly supervised manner to learn the apathetic behaviour in order to address these challenges. Features from facial expressions, action units, facial landmarks and audio signals are extracted for training. The fusion of multiple modalities in a weakly supervised method achieves 75.71% accuracy for apathy detection in elderly people. The experiments show that multimodal fusion is able to leverage on the presence of complimentary information across different modalities.

Original languageEnglish
Title of host publication2020 International Joint Conference on Neural Networks (IJCNN), 2020 Conference Proceedings2020 International Joint Conference on Neural Networks, IJCNN 2020 - Proceedings
EditorsAsim Roy
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Number of pages7
ISBN (Electronic)9781728169262
ISBN (Print)9781728169279
Publication statusPublished - 2020
EventIEEE International Joint Conference on Neural Networks 2020 - Virtual, Glasgow, United Kingdom
Duration: 19 Jul 202024 Jul 2020 (Proceedings) (Website)


ConferenceIEEE International Joint Conference on Neural Networks 2020
Abbreviated titleIJCNN 2020
Country/TerritoryUnited Kingdom
CityVirtual, Glasgow
Internet address


  • Apathy detection
  • Digital health
  • Emotion recognition
  • Multiple instance learning

Cite this