Joint facial expression recognition and intensity estimation based on weighted votes of image sequences

Siti Khairuni Amalina Kamarol, Mohamed Hisham Jaward, Heikki Kälviäinen, Jussi Parkkinen, Rajendran Parthiban

    Research output: Contribution to journalArticleResearchpeer-review

    24 Citations (Scopus)


    Facial behavior consists of dynamically changing properties of facial features as a result of muscle activation. Facial behavior analysis is a challenging problem due to complexity of emotions and variability of the facial expressions associated with the emotions. Most facial expression recognition systems attempt to recognize facial expressions without taking into account the intensity of the expressions. In this paper, a novel framework for facial expression recognition and intensity estimation with low computational complexity requirement is proposed. The algorithm constructs a representation of facial features based on a weighted voting scheme and employs Hidden Markov Models to classify an input video into one of the six basic expressions, namely anger, disgust, fear, happiness, sadness, and surprise. The temporal segments, neutral, onset, and apex, of an expression are then obtained by means of a change-point detector. Evaluations on subject-independent analysis was conducted using Cohn-Kanade dataset and Beihang University facial expression datasets. The proposed approach has demonstrated a superior performance in recognizing facial expressions and estimating expression intensities.

    Original languageEnglish
    Pages (from-to)25-32
    Number of pages8
    JournalPattern Recognition Letters
    Publication statusPublished - 1 Jun 2017


    • Change-point detection
    • Computer vision
    • Facial expression recognition
    • Hidden markov models
    • Video-based facial expression intensity analysis
    • Voting scheme

    Cite this