Abstract
Speech-driven 3D facial animation has been widely studied, yet there is still a gap to achieving realism and vividness due to the highly ill-posed nature and scarcity of audio-visual data. Existing works typically formulate the cross-modal mapping into a regression task, which suffers from the regression-to-mean problem leading to over-smoothed facial motions. In this paper, we propose to cast speech-driven facial animation as a code query task in a finite proxy space of the learned codebook, which effectively promotes the vividness of the generated motions by reducing the cross-modal mapping uncertainty. The codebook is learned by self-reconstruction over real facial motions and thus embedded with realistic facial motion priors. Over the discrete motion space, a temporal autoregressive model is employed to sequentially synthesize facial motions from the input speech signal, which guarantees lip-sync as well as plausible facial expressions. We demonstrate that our approach outperforms current state-of-the-art methods both qualitatively and quantitatively. Also, a user study further justifies our superiority in perceptual quality. Code and video demo are available at https://doubiiu.github.io/projects/codetalker.
Original language | English |
---|---|
Title of host publication | Proceedings - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023 |
Editors | Eric Mortensen |
Place of Publication | Piscataway NJ USA |
Publisher | IEEE, Institute of Electrical and Electronics Engineers |
Number of pages | 11 |
ISBN (Electronic) | 9798350301298 |
ISBN (Print) | 9798350301304 |
DOIs | |
Publication status | Published - 2023 |
Externally published | Yes |
Event | IEEE Conference on Computer Vision and Pattern Recognition 2023 - Vancouver, Canada Duration: 18 Jun 2023 → 22 Jun 2023 https://cvpr2023.thecvf.com/ (Website) https://openaccess.thecvf.com/CVPR2023?day=all (Proceedings) https://ieeexplore.ieee.org/xpl/conhome/10203037/proceeding (Proceedings) https://cvpr2023.thecvf.com/Conferences/2023 (Website) |
Conference
Conference | IEEE Conference on Computer Vision and Pattern Recognition 2023 |
---|---|
Abbreviated title | CVPR 2023 |
Country/Territory | Canada |
City | Vancouver |
Period | 18/06/23 → 22/06/23 |
Internet address |
|