MakeItTalk: Speaker-Aware Talking-Head Animation

Abstract

We present a method that generates expressive talking heads from a single facial image with audio as the only input. In contrast to previous approaches that attempt to learn direct mappings from audio to raw pixels or points for creating talking faces, our method first disentangles the content and speaker information in the input audio signal. The audio content robustly controls the motion of lips and nearby facial regions, while the speaker information determines the specifics of facial expressions and the rest of the talking head dynamics. Another key component of our method is the prediction of facial landmarks reflecting speaker-aware dynamics. Based on this intermediate representation, our method is able to synthesize photorealistic videos of entire talking heads with full range of motion and also animate artistic paintings, sketches, 2D cartoon characters, Japanese mangas, stylized caricatures in a single unified framework. We present extensive quantitative and qualitative evaluation of our method, in addition to user studies, demonstrating generated talking heads of significantly higher quality compared to prior state-of-the-art.

SIGGRAPH Asia 2020, 15 pages, 13 figures

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• VoxCeleb2: Deep Speaker Recognition
• How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)
• AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
• First Order Motion Model for Image Animation
• Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss
• Improvising Linguistic Style: Social and Affective Bases for Agent Personality
• VisemeNet: Audio-Driven Animator-Centric Speech Animation

Cited by in corpus (26)

• Imitating Arbitrary Talking Style for Realistic Audio-DrivenTalking Face Synthesis
• Deep Person Generation: A Survey from the Perspective of Face, Pose and Cloth Synthesis
• Towards Realistic Visual Dubbing with Heterogeneous Sources
• DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder
• GaussianTalker: Speaker-specific Talking Head Synthesis via 3D Gaussian Splatting
• Talking Head from Speech Audio using a Pre-trained Image Generator
• Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation
• Wakey-Wakey: Animate Text by Mimicking Characters in a GIF
• Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separation
• Audio-Driven Dubbing for User Generated Contents via Style-Aware Semi-Parametric Synthesis
• Human Motion Video Generation: A Survey
• Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation
• Talking Head Generation with Audio and Speech Related Facial Action Units
• Audio-Driven Emotional Video Portraits
• Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion
• Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation
• Controllable Person Image Synthesis with Spatially-Adaptive Warped Normalization
• Intelligent Video Editing: Incorporating Modern Talking Face Generation Algorithms in a Video Editor
• 3D-TalkEmo: Learning to Synthesize 3D Emotional Talking Head
• Everything's Talkin': Pareidolia Face Reenactment
• Extreme-scale Talking-Face Video Upsampling with Audio-Visual Priors
• Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functions
• Model See Model Do: Speech-Driven Facial Animation with Style Control
• PTeacher: a Computer-Aided Personalized Pronunciation Training System with Exaggerated Audio-Visual Corrective Feedback
• AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person
• Sparse to Dense Motion Transfer for Face Image Animation