Real-time 3D face-eye performance capture of a person wearing vr headset

Guoxian Song, Jianfei Cai, Tat-Jen Cham, Jianmin Zheng, Juyong Zhang, Henry Fuchs

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

6 Citations (Scopus)


Teleconference or telepresence based on virtual reality (VR) head-mount display (HMD) device is a very interesting and promising application since HMD can provide immersive feelings for users. However, in order to facilitate face-to-face communications for HMD users, real-time 3D facial performance capture of a person wearing HMD is needed, which is a very challenging task due to the large occlusion caused by HMD. The existing limited solutions are very complex either in setting or in approach as well as lacking the performance capture of 3D eye gaze movement. In this paper, we propose a convolutional neural network (CNN) based solution for real-time 3D face-eye performance capture of HMD users without complex modification to devices. To address the issue of lacking training data, we generate massive pairs of HMD face-label dataset by data synthesis as well as collecting VR-IR eye dataset from multiple subjects. Then, we train a dense-fitting network for facial region and an eye gaze network to regress 3D eye model parameters. Extensive experimental results demonstrate that our system can efficiently and effectively produce in real time a vivid personalized 3D avatar with the correct identity, pose, expression and eye motion corresponding to the HMD user.

Original languageEnglish
Title of host publicationMM’18 - Proceedings of the 2018 ACM Multimedia Conference
EditorsHyeran Byun, Chang Wen Chen, Rainer Lienhart, Tao Mei
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages9
ISBN (Electronic)9781450356657
Publication statusPublished - 2018
Externally publishedYes
EventACM International Conference on Multimedia 2018 - Seoul, Korea, South
Duration: 22 Oct 201826 Oct 2018
Conference number: 26th


ConferenceACM International Conference on Multimedia 2018
Abbreviated titleMM 2018
Country/TerritoryKorea, South
Internet address


  • 3D facial reconstruction
  • Gaze estimation
  • HMDs

Cite this