Real-time shadow-aware portrait relighting in virtual backgrounds for realistic telepresence

Guoxian Song, Tat Jen Cham, Jianfei Cai, Jianmin Zheng

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

3 Citations (Scopus)

Abstract

While using virtual backgrounds has recently become a very popular feature in videoconferencing, there often exists a jarring mismatch between the lighting of the user and the illumination condition of the virtual background. Existing portrait relighting methods can alleviate the problem, but do not have the capacity to deal with difficult shadow effects. In this paper, we present a new shadow-aware portrait relighting system that can relight an input portrait to be consistent with a given desired background image with shadow effects. Our system consists of four major components: portrait neutralization, illumination estimation, shadow generation and hierarchical neural rendering, which are all based on deep neural networks, and the whole system is end-to-end trainable. In addition, we created a large-scale photorealistic synthetic dataset with shadow, illumination and depth annotations for training, which allows our model to generalize well to real images. The extensive experiments demonstrate that our shadow-aware relight system outperforms the state-of-the-art portrait relighting solutions in terms of producing more lighting-consistent relighted images with shadow effects.

Original languageEnglish
Title of host publicationProceedings - 2022 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2022
EditorsHenry Duh, Ian Williams, Jens Grubert, J. Adam Jones, Jianmin Zheng
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages729-738
Number of pages10
ISBN (Electronic)9781665453257
ISBN (Print)9781665453264
DOIs
Publication statusPublished - 2022
EventIEEE International Symposium on Mixed and Augmented Reality 2022 - Singapore, Singapore
Duration: 17 Oct 202221 Oct 2022
Conference number: 21st
https://ieeexplore.ieee.org/xpl/conhome/9994794/proceeding (Proceedings)
https://ismar2022.org/ (Website)

Publication series

NameProceedings - 2022 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2022
PublisherIEEE, Institute of Electrical and Electronics Engineers
ISSN (Print)1554-7868

Conference

ConferenceIEEE International Symposium on Mixed and Augmented Reality 2022
Abbreviated titleISMAR 2022
Country/TerritorySingapore
CitySingapore
Period17/10/2221/10/22
Internet address

Keywords

  • Neural Rendering
  • Relighting
  • Shadow Generation

Cite this