Generative Adversarial Talking Head: Bringing Portraits to Life with a Weakly Supervised Neural Network

Abstract

This paper presents Generative Adversarial Talking Head (GATH), a novel deep generative neural network that enables fully automatic facial expression synthesis of an arbitrary portrait with continuous action unit (AU) coefficients. Specifically, our model directly manipulates image pixels to make the unseen subject in the still photo express various emotions controlled by values of facial AU coefficients, while maintaining her personal characteristics, such as facial geometry, skin color and hair style, as well as the original surrounding background. In contrast to prior work, GATH is purely data-driven and it requires neither a statistical face model nor image processing tricks to enact facial deformations. Additionally, our model is trained from unpaired data, where the input image, with its auxiliary identity label taken from abundance of still photos in the wild, and the target frame are from different persons. In order to effectively learn such model, we propose a novel weakly supervised adversarial learning framework that consists of a generator, a discriminator, a classifier and an action unit estimator. Our work gives rise to template-and-target-free expression editing, where still faces can be effortlessly animated with arbitrary AU coefficients provided by the user.

Fix typos, add youtube link of supplementary video

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Auto-Encoding Variational Bayes
• Conditional Generative Adversarial Nets
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
• Triple Generative Adversarial Nets
• Automatic Face Reenactment
• Semantic Facial Expression Editing using Autoencoded Flow
• Conditional Adversarial Synthesis of 3D Facial Action Units
• End-to-end Learning for 3D Facial Animation from Raw Waveforms of Speech

Cited by in corpus (8)

• DeepFakes: a New Threat to Face Recognition? Assessment and Detection
• Deepfake Video Detection Using Convolutional Vision Transformer
• Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics
• Deepfake Detection using Spatiotemporal Convolutional Networks
• Everybody's Talkin': Let Me Talk as You Want
• Adversarial Training in Affective Computing and Sentiment Analysis: Recent Advances and Perspectives
• Vulnerability of Face Recognition to Deep Morphing
• FLNet: Landmark Driven Fetching and Learning Network for Faithful Talking Facial Animation Synthesis