Audio-visual weakly supervised approach for apathy detection in the elderly

Apathy is manifested as lack of feelings or emotions in several neurological and psychological disorders. Hence, directly impairing the display of emotion through facial expressions and speech. Current practices of prediction of apathy heavily rely on clinical diagnosis, an expert interviewing a patient or reports from patients' family members. The dependence on an expert and the human bias in its examination results in under-diagnosis of condition. In this paper, a multimodal multi-instance learning based method is proposed for automatic apathy detection. There are several challenges present while automating the process. Some of which are - recognizing emotions in elderly people, correct identification of emotions in a conversation and identifying differences between emotional responses from apathetic and non apathetic cohorts. The proposed method uses the audio and visual information in a weakly supervised manner to learn the apathetic behaviour in order to address these challenges. Features from facial expressions, action units, facial landmarks and audio signals are extracted for training. The fusion of multiple modalities in a weakly supervised method achieves 75.71% accuracy for apathy detection in elderly people. The experiments show that multimodal fusion is able to leverage on the presence of complimentary information across different modalities.