Automatic group affect analysis in images via visual attribute and feature networks

Shreya Ghosh, Abhinav Dhall, Nicu Sebe

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

5 Citations (Scopus)

Abstract

This paper proposes a pipeline for automatic group-level affect analysis. A deep neural network-based approach, which leverages on the facial-expression information, scene information and a high-level facial visual attribute information is proposed. A capsule network-based architecture is used to predict the facial expression. Transfer learning is used on Inception-V3 to extract global image-based features which contain scene information. Another network is trained for inferring the facial attributes of the group members. Further, these attributes are pooled at a group-level to train a network for inferring the group-level affect. The facial attribute prediction network, although is simple yet, is effective and generates result comparable to the state-of-the-art methods. Later, model integration is performed from the three channels. The experiments show the effectiveness of the proposed techniques on three 'in the wild' databases: Group Affect Database, HAPPEI and UCLA-Protest database.

Original languageEnglish
Title of host publication2018 IEEE International Conference on Image Processing - Proceedings
Subtitle of host publicationOctober 7–10, 2018 Megaron Athens International Conference Centre Athens, Greece
EditorsNikolaos Boulgouris, Lisimachos P. Kondi
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages1967-1971
Number of pages5
ISBN (Electronic)9781479970612
ISBN (Print)9781479970629
DOIs
Publication statusPublished - 2018
Externally publishedYes
EventIEEE International Conference on Image Processing 2018 - Athens, Greece
Duration: 7 Oct 201810 Oct 2018
Conference number: 25th
https://2018.ieeeicip.org/

Conference

ConferenceIEEE International Conference on Image Processing 2018
Abbreviated titleICIP 2018
CountryGreece
CityAthens
Period7/10/1810/10/18
Internet address

Keywords

  • Group level affect recognition

Cite this

Ghosh, S., Dhall, A., & Sebe, N. (2018). Automatic group affect analysis in images via visual attribute and feature networks. In N. Boulgouris, & L. P. Kondi (Eds.), 2018 IEEE International Conference on Image Processing - Proceedings : October 7–10, 2018 Megaron Athens International Conference Centre Athens, Greece (pp. 1967-1971). [8451242] Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ICIP.2018.8451242
Ghosh, Shreya ; Dhall, Abhinav ; Sebe, Nicu. / Automatic group affect analysis in images via visual attribute and feature networks. 2018 IEEE International Conference on Image Processing - Proceedings : October 7–10, 2018 Megaron Athens International Conference Centre Athens, Greece. editor / Nikolaos Boulgouris ; Lisimachos P. Kondi. Piscataway NJ USA : IEEE, Institute of Electrical and Electronics Engineers, 2018. pp. 1967-1971
@inproceedings{b4efc966431e4c9d99ea4d018dd26b75,
title = "Automatic group affect analysis in images via visual attribute and feature networks",
abstract = "This paper proposes a pipeline for automatic group-level affect analysis. A deep neural network-based approach, which leverages on the facial-expression information, scene information and a high-level facial visual attribute information is proposed. A capsule network-based architecture is used to predict the facial expression. Transfer learning is used on Inception-V3 to extract global image-based features which contain scene information. Another network is trained for inferring the facial attributes of the group members. Further, these attributes are pooled at a group-level to train a network for inferring the group-level affect. The facial attribute prediction network, although is simple yet, is effective and generates result comparable to the state-of-the-art methods. Later, model integration is performed from the three channels. The experiments show the effectiveness of the proposed techniques on three 'in the wild' databases: Group Affect Database, HAPPEI and UCLA-Protest database.",
keywords = "Group level affect recognition",
author = "Shreya Ghosh and Abhinav Dhall and Nicu Sebe",
year = "2018",
doi = "10.1109/ICIP.2018.8451242",
language = "English",
isbn = "9781479970629",
pages = "1967--1971",
editor = "Boulgouris, {Nikolaos } and {P. Kondi}, {Lisimachos }",
booktitle = "2018 IEEE International Conference on Image Processing - Proceedings",
publisher = "IEEE, Institute of Electrical and Electronics Engineers",
address = "United States of America",

}

Ghosh, S, Dhall, A & Sebe, N 2018, Automatic group affect analysis in images via visual attribute and feature networks. in N Boulgouris & L P. Kondi (eds), 2018 IEEE International Conference on Image Processing - Proceedings : October 7–10, 2018 Megaron Athens International Conference Centre Athens, Greece., 8451242, IEEE, Institute of Electrical and Electronics Engineers, Piscataway NJ USA, pp. 1967-1971, IEEE International Conference on Image Processing 2018, Athens, Greece, 7/10/18. https://doi.org/10.1109/ICIP.2018.8451242

Automatic group affect analysis in images via visual attribute and feature networks. / Ghosh, Shreya; Dhall, Abhinav; Sebe, Nicu.

2018 IEEE International Conference on Image Processing - Proceedings : October 7–10, 2018 Megaron Athens International Conference Centre Athens, Greece. ed. / Nikolaos Boulgouris; Lisimachos P. Kondi. Piscataway NJ USA : IEEE, Institute of Electrical and Electronics Engineers, 2018. p. 1967-1971 8451242.

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

TY - GEN

T1 - Automatic group affect analysis in images via visual attribute and feature networks

AU - Ghosh, Shreya

AU - Dhall, Abhinav

AU - Sebe, Nicu

PY - 2018

Y1 - 2018

N2 - This paper proposes a pipeline for automatic group-level affect analysis. A deep neural network-based approach, which leverages on the facial-expression information, scene information and a high-level facial visual attribute information is proposed. A capsule network-based architecture is used to predict the facial expression. Transfer learning is used on Inception-V3 to extract global image-based features which contain scene information. Another network is trained for inferring the facial attributes of the group members. Further, these attributes are pooled at a group-level to train a network for inferring the group-level affect. The facial attribute prediction network, although is simple yet, is effective and generates result comparable to the state-of-the-art methods. Later, model integration is performed from the three channels. The experiments show the effectiveness of the proposed techniques on three 'in the wild' databases: Group Affect Database, HAPPEI and UCLA-Protest database.

AB - This paper proposes a pipeline for automatic group-level affect analysis. A deep neural network-based approach, which leverages on the facial-expression information, scene information and a high-level facial visual attribute information is proposed. A capsule network-based architecture is used to predict the facial expression. Transfer learning is used on Inception-V3 to extract global image-based features which contain scene information. Another network is trained for inferring the facial attributes of the group members. Further, these attributes are pooled at a group-level to train a network for inferring the group-level affect. The facial attribute prediction network, although is simple yet, is effective and generates result comparable to the state-of-the-art methods. Later, model integration is performed from the three channels. The experiments show the effectiveness of the proposed techniques on three 'in the wild' databases: Group Affect Database, HAPPEI and UCLA-Protest database.

KW - Group level affect recognition

UR - http://www.scopus.com/inward/record.url?scp=85061780622&partnerID=8YFLogxK

U2 - 10.1109/ICIP.2018.8451242

DO - 10.1109/ICIP.2018.8451242

M3 - Conference Paper

SN - 9781479970629

SP - 1967

EP - 1971

BT - 2018 IEEE International Conference on Image Processing - Proceedings

A2 - Boulgouris, Nikolaos

A2 - P. Kondi, Lisimachos

PB - IEEE, Institute of Electrical and Electronics Engineers

CY - Piscataway NJ USA

ER -

Ghosh S, Dhall A, Sebe N. Automatic group affect analysis in images via visual attribute and feature networks. In Boulgouris N, P. Kondi L, editors, 2018 IEEE International Conference on Image Processing - Proceedings : October 7–10, 2018 Megaron Athens International Conference Centre Athens, Greece. Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers. 2018. p. 1967-1971. 8451242 https://doi.org/10.1109/ICIP.2018.8451242