Model See Model Do: Speech-Driven Facial Animation with Style Control

Abstract

Speech-driven 3D facial animation plays a key role in applications such as virtual avatars, gaming, and digital content creation. While existing methods have made significant progress in achieving accurate lip synchronization and generating basic emotional expressions, they often struggle to capture and effectively transfer nuanced performance styles. We propose a novel example-based generation framework that conditions a latent diffusion model on a reference style clip to produce highly expressive and temporally coherent facial animations. To address the challenge of accurately adhering to the style reference, we introduce a novel conditioning mechanism called style basis, which extracts key poses from the reference and additively guides the diffusion generation process to fit the style without compromising lip synchronization quality. This approach enables the model to capture subtle stylistic cues while ensuring that the generated animations align closely with the input speech. Extensive qualitative, quantitative, and perceptual evaluations demonstrate the effectiveness of our method in faithfully reproducing the desired style while achieving superior lip synchronization across various speech scenarios.

10 pages, 7 figures, SIGGRAPH Conference Papers '25

References in corpus (24)

• Denoising Diffusion Probabilistic Models
• Classifier-Free Diffusion Guidance
• High-Resolution Image Synthesis with Latent Diffusion Models
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• MakeItTalk: Speaker-Aware Talking-Head Animation
• Human Motion Diffusion Model
• Emotional Speech-Driven Animation with Content-Emotion Disentanglement
• Transflower: probabilistic autoregressive dance generation with multimodal attention
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
• Daft-Exprt: Cross-Speaker Prosody Transfer on Any Text for Expressive Speech Synthesis
• EMOCA: Emotion Driven Monocular Face Capture and Animation
• Learning an Animatable Detailed 3D Face Model from In-The-Wild Images
• FaceFormer: Speech-Driven 3D Facial Animation with Transformers
• ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech
• CelebV-HQ: A Large-Scale Video Facial Attributes Dataset
• SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation
• EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation
• VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time
• TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Styles
• Learning latent representations for style control and transfer in end-to-end speech synthesis
• Optimizing Diffusion Noise Can Serve As Universal Motion Priors
• Imitator: Personalized Speech-driven 3D Facial Animation
• Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidance
• CelebV-Text: A Large-Scale Facial Text-Video Dataset