Audio-Driven Emotional Video Portraits
arXiv:2104.07452
Abstract
Despite previous success in generating audio-driven talking heads, most of the previous studies focus on the correlation between speech content and the mouth shape. Facial emotion, which is one of the most important features on natural human faces, is always neglected in their methods. In this work, we present Emotional Video Portraits (EVP), a system for synthesizing high-quality video portraits with vivid emotional dynamics driven by audios. Specifically, we propose the Cross-Reconstructed Emotion Disentanglement technique to decompose speech into two decoupled spaces, i.e., a duration-independent emotion space and a duration dependent content space. With the disentangled features, dynamic 2D emotional facial landmarks can be deduced. Then we propose the Target-Adaptive Face Synthesis technique to generate the final high-quality video portraits, by bridging the gap between the deduced landmarks and the natural head poses of target videos. Extensive experiments demonstrate the effectiveness of our method both qualitatively and quantitatively.
arXiv commentAccepted by CVPR2021
References in corpus (6)
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- MakeItTalk: Speaker-Aware Talking-Head Animation
- DRIT++: Diverse Image-to-Image Translation via Disentangled Representations
- ExprGAN: Facial Expression Editing with Controllable Expression Intensity
- Everybody's Talkin': Let Me Talk as You Want
- What comprises a good talking-head video generation?: A Survey and Benchmark