Talking-head Generation with Rhythmic Head Motion
arXiv:2007.08547
Abstract
When people deliver a speech, they naturally move heads, and this rhythmic head motion conveys prosodic information. However, generating a lip-synced video while moving head naturally is challenging. While remarkably successful, existing works either generate still talkingface videos or rely on landmark/video frames as sparse/dense mapping guidance to generate head movements, which leads to unrealistic or uncontrollable video synthesis. To overcome the limitations, we propose a 3D-aware generative network along with a hybrid embedding module and a non-linear composition module. Through modeling the head motion and facial expressions1 explicitly, manipulating 3D animation carefully, and embedding reference images dynamically, our approach achieves controllable, photo-realistic, and temporally coherent talking-head videos with natural head movements. Thoughtful experiments on several standard benchmarks demonstrate that our method achieves significantly better results than the state-of-the-art methods in both quantitative and qualitative comparisons. The code is available on https://github.com/ lelechen63/Talking-head-Generation-with-Rhythmic-Head-Motion.
References in corpus (6)
- Adam: A Method for Stochastic Optimization
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- VoxCeleb2: Deep Speaker Recognition
- LRS3-TED: a large-scale dataset for visual speech recognition
- Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss
- Talking Face Generation by Conditional Recurrent Adversarial Network