A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony in Talking Head Generation

Abstract

Animating still face images with deep generative models using a speech input signal is an active research topic and has seen important recent progress.However, much of the effort has been put into lip syncing and rendering quality while the generation of natural head motion, let alone the audio-visual correlation between head motion and speech, has often been neglected.In this work, we propose a multi-scale audio-visual synchrony loss and a multi-scale autoregressive GAN to better handle short and long-term correlation between speech and the dynamics of the head and lips.In particular, we train a stack of syncer models on multimodal input pyramids and use these models as guidance in a multi-scale generator network to produce audio-aligned motion unfolding over diverse time scales.Both the pyramid of audio-visual syncers and the generative models are trained in a low-dimensional space that fully preserves dynamics cues.The experiments show significant improvements over the state-of-the-art in head motion dynamics quality and especially in multi-scale audio-visual synchrony on a collection of benchmark datasets.

References in corpus (12)

• Representation Learning with Contrastive Predictive Coding
• VoxCeleb2: Deep Speaker Recognition
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)
• A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild
• Deep Audio-Visual Speech Recognition
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose
• Geometric GAN
• Human Motion Modeling using DVGANs
• Arbitrary Talking Face Generation via Attentional Audio-Visual Coherence Learning
• FaceFormer: Speech-Driven 3D Facial Animation with Transformers