Speech-Driven Facial Reenactment Using Conditional Generative Adversarial Networks

Abstract

We present a novel approach to generating photo-realistic images of a face with accurate lip sync, given an audio input. By using a recurrent neural network, we achieved mouth landmarks based on audio features. We exploited the power of conditional generative adversarial networks to produce highly-realistic face conditioned on a set of landmarks. These two networks together are capable of producing a sequence of natural faces in sync with an input audio track.

arXiv commentSubmitted for ECCV 2018

References in corpus (6)

• Conditional Generative Adversarial Nets
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Autoencoding beyond pixels using a learned similarity metric
• Semi-Latent GAN: Learning to generate and modify facial images from attributes
• AENet: Learning Deep Audio Features for Video Analysis

Cited by in corpus (5)

• Trends in deep learning for medical hyperspectral image analysis
• Arbitrary Talking Face Generation via Attentional Audio-Visual Coherence Learning
• Deep Audio-Visual Learning: A Survey
• Animating Face using Disentangled Audio Representations
• Lets Play Music: Audio-driven Performance Video Generation