One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing
arXiv:2011.15126
Abstract
We propose a neural talking-head video synthesis model and demonstrate its application to video conferencing. Our model learns to synthesize a talking-head video using a source image containing the target person's appearance and a driving video that dictates the motion in the output. Our motion is encoded based on a novel keypoint representation, where the identity-specific and motion-related information is decomposed unsupervisedly. Extensive experimental validation shows that our model outperforms competing methods on benchmark datasets. Moreover, our compact keypoint representation enables a video conferencing system that achieves the same visual quality as the commercial H.264 standard while only using one-tenth of the bandwidth. Besides, we show our keypoint representation allows the user to rotate the head during synthesis, which is useful for simulating face-to-face video conferencing experiences.
arXiv commentCVPR 2021 camera ready (oral). Our project page can be found at https://nvlabs.github.io/face-vid2vid
References in corpus (15)
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- VoxCeleb2: Deep Speaker Recognition
- Toward Multimodal Image-to-Image Translation
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
- Semantic Image Synthesis with Spatially-Adaptive Normalization
- First Order Motion Model for Image Animation
- Image Super-Resolution Using Deep Convolutional Networks
- Few-Shot Unsupervised Image-to-Image Translation
- Deferred Neural Rendering: Image Synthesis using Neural Textures
- Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss
- Deep Image Spatial Transformation for Person Image Generation
- Foreground-aware Image Inpainting
- Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images
- Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning
- ReenactGAN: Learning to Reenact Faces via Boundary Transfer