Automatic Face Reenactment

Abstract

We propose an image-based, facial reenactment system that replaces the face of an actor in an existing target video with the face of a user from a source video, while preserving the original target performance. Our system is fully automatic and does not require a database of source expressions. Instead, it is able to produce convincing reenactment results from a short source video captured with an off-the-shelf camera, such as a webcam, where the user performs arbitrary facial gestures. Our reenactment pipeline is conceived as part image retrieval and part face transfer: The image retrieval is based on temporal clustering of target frames and a novel image matching metric that combines appearance and motion to select candidate frames from the source video, while the face transfer uses a 2D warping strategy that preserves the user's identity. Our system excels in simplicity as it does not rely on a 3D face model, it is robust under head motion and does not require the source and target performance to be similar. We show convincing reenactment results for videos that we recorded ourselves and for low-quality footage taken from the Internet.

Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (8 pages)

Cited by in corpus (34)

• MesoNet: a Compact Facial Video Forgery Detection Network
• The Creation and Detection of Deepfakes: A Survey
• HeadOn: Real-time Reenactment of Human Portrait Videos
• Deep Learning Based Single Sample Per Person Face Recognition: A Survey
• Deepfake Detection using Spatiotemporal Convolutional Networks
• DeepRhythm: Exposing DeepFakes with Attentional Visual Heartbeat Rhythms
• Head2Head++: Deep Facial Attributes Re-Targeting
• FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality
• Face Transfer with Generative Adversarial Network
• Generative Adversarial Talking Head: Bringing Portraits to Life with a Weakly Supervised Neural Network
• ReenactGAN: Learning to Reenact Faces via Boundary Transfer
• Explicit Facial Expression Transfer via Fine-Grained Representations
• Audio-Driven Dubbing for User Generated Contents via Style-Aware Semi-Parametric Synthesis
• Exploiting Human Social Cognition for the Detection of Fake and Fraudulent Faces via Memory Networks
• Impact of Facial Tattoos and Paintings on Face Recognition Systems
• How Do the Hearts of Deep Fakes Beat? Deep Fake Source Detection via Interpreting Residuals with Biological Signals
• Spatiotemporal Inconsistency Learning for DeepFake Video Detection
• All-In-One: Facial Expression Transfer, Editing and Recognition Using A Single Network
• Generalized Zero and Few-Shot Transfer for Facial Forgery Detection
• Unconstrained Facial Expression Transfer using Style-based Generator
• Neural 3D Clothes Retargeting from a Single Image
• CNN-based Real-time Dense Face Reconstruction with Inverse-rendered Photo-realistic Face Images
• Task-agnostic Temporally Consistent Facial Video Editing
• What Makes Kevin Spacey Look Like Kevin Spacey
• A Neural Virtual Anchor Synthesizer based on Seq2Seq and GAN Models
• Everything's Talkin': Pareidolia Face Reenactment
• Iterative Text-based Editing of Talking-heads Using Neural Retargeting
• Voice2Mesh: Cross-Modal 3D Face Model Generation from Voices
• Face2Face: Real-time Face Capture and Reenactment of RGB Videos
• Convolutional Humanoid Animation via Deformation
• Head2HeadFS: Video-based Head Reenactment with Few-shot Learning
• Towards Fine-grained 3D Face Dense Registration: An Optimal Dividing and Diffusing Method
• UniFaceGAN: A Unified Framework for Temporally Consistent Facial Video Editing
• Exploring Adversarial Fake Images on Face Manifold