Jointly predicting affective and mental health scores using deep neural networks of visual cues on the web

Hung Nguyen, Van Nguyen, Thin Nguyen, Mark E. Larsen, Bridianne O’Dea, Duc Thanh Nguyen, Trung Le, Dinh Phung, Svetha Venkatesh, Helen Christensen

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

1 Citation (Scopus)


Despite the range of studies examining the relationship between mental health and social media data, not all prior studies have validated the social media markers against “ground truth”, or validated psychiatric information, in general community samples. Instead, researchers have approximated psychiatric diagnosis using user statements such as “I have been diagnosed as X”. Without “ground truth”, the value of predictive algorithms is highly questionable and potentially harmful. In addition, for social media data, whilst linguistic features have been widely identified as strong markers of mental health disorders, little is known about non-textual features on their links with the disorders. The current work is a longitudinal study during which participants’ mental health data, consisting of depression and anxiety scores, were collected fortnightly with a validated, diagnostic, clinical measure. Also, datasets with labels relevant to mental health scores, such as emotional scores, are also employed to improve the performance in prediction of mental health scores. This work introduces a deep neural network-based method integrating sub-networks on predicting affective scores and mental health outcomes from images. Experimental results have shown that in the both predictions of emotion and mental health scores, (1) deep features majorly outperform handcrafted ones and (2) the proposed network achieves better performance compared with separate networks.

Original languageEnglish
Title of host publicationWeb Information Systems Engineering – WISE 2018
Subtitle of host publication19th International Conference Dubai, United Arab Emirates, November 12–15, 2018 Proceedings, Part II
EditorsHakim Hacid, Wojciech Cellary, Hua Wang, Hye-Young Paik, Rui Zhou
Place of PublicationCham Switzerland
Number of pages11
ISBN (Electronic)9783030029258
ISBN (Print)9783030029241
Publication statusPublished - 2018
EventInternational Conference on Web Information Systems Engineering 2018 - Dubai, United Arab Emirates
Duration: 12 Nov 201815 Nov 2018
Conference number: 19th (Proceedings)

Publication series

NameLecture Notes in Computer Science
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


ConferenceInternational Conference on Web Information Systems Engineering 2018
Abbreviated titleWISE 2018
Country/TerritoryUnited Arab Emirates
Internet address


  • Behavioral monitoring
  • Deep learning
  • Health analytics
  • Mental health
  • Social media
  • Visual features

Cite this