Can we see more? Joint frontalization and hallucination of unaligned tiny faces

Xin Yu, Fatemeh Shiri, Bernard Ghanem, Fatih Porikli

Research output: Contribution to journalArticleResearchpeer-review

3 Citations (Scopus)

Abstract

In popular TV programs (such as CSI), a very low-resolution face image of a person, who is not even looking at the camera in many cases, is digitally super-resolved to a degree that suddenly the person's identity is made visible and recognizable. Of course, we suspect that this is merely a cinematographic special effect and such a magical transformation of a single image is not technically possible. Or, is it? In this paper, we push the boundaries of super-resolving (hallucinating to be more accurate) a tiny, non-frontal face image to understand how much of this is possible by leveraging the availability of large datasets and deep networks. To this end, we introduce a novel Transformative Adversarial Neural Network (TANN) to jointly frontalize very-low resolution (i.e., 16 × 16 pixels) out-of-plane rotated face images (including profile views) and aggressively super-resolve them (8×), regardless of their original poses and without using any 3D information. TANN is composed of two components: a transformative upsampling network which embodies encoding, spatial transformation and deconvolutional layers, and a discriminative network that enforces the generated high-resolution frontal faces to lie on the same manifold as real frontal face images. We evaluate our method on a large set of synthesized non-frontal face images to assess its reconstruction performance. Extensive experiments demonstrate that TANN generates both qualitatively and quantitatively superior results achieving over 4 dB improvement over the state-of-the-art.

Original languageEnglish
Pages (from-to)2148-2164
Number of pages17
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Volume42
Issue number9
DOIs
Publication statusPublished - 1 Sep 2020
Externally publishedYes

Keywords

  • Face
  • face frontalization
  • hallucination
  • super-resolution

Cite this