First Order Motion Model for Image Animation
arXiv:2003.00196
Abstract
Image animation consists of generating a video sequence so that an object in a source image is animated according to the motion of a driving video. Our framework addresses this problem without using any annotation or prior information about the specific object to animate. Once trained on a set of videos depicting objects of the same category (e.g. faces, human bodies), our method can be applied to any object of this class. To achieve this, we decouple appearance and motion information using a self-supervised formulation. To support complex motions, we use a representation consisting of a set of learned keypoints along with their local affine transformations. A generator network models occlusions arising during target motions and combines the appearance extracted from the source image and the motion derived from the driving video. Our framework scores best on diverse benchmarks and on a variety of object categories. Our source code is publicly available.
NeurIPS 2019
Cited by in corpus (47)
- MakeItTalk: Speaker-Aware Talking-Head Animation
- One Detector to Rule Them All: Towards a General Deepfake Attack Detection Framework
- Deep Spatial Transformation for Pose-Guided Person Image Generation and Animation
- SofGAN: A Portrait Image Generator with Dynamic Styling
- Head2Head++: Deep Facial Attributes Re-Targeting
- Portrait Neural Radiance Fields from a Single Image
- Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
- F3A-GAN: Facial Flow for Face Animation with Generative Adversarial Networks
- Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation
- One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing
- PhotoApp: Photorealistic Appearance Editing of Head Portraits
- Am I a Real or Fake Celebrity? Measuring Commercial Face Recognition Web APIs under Deepfake Impersonation Attack
- Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation
- Image Comes Dancing with Collaborative Parsing-Flow Video Synthesis
- PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering
- Human Motion Transfer from Poses in the Wild
- One-Shot Identity-Preserving Portrait Reenactment
- Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation
- Intelligent Video Editing: Incorporating Modern Talking Face Generation Algorithms in a Video Editor
- Robust Pose Transfer with Dynamic Details using Neural Video Rendering
- Move As You Like: Image Animation in E-Commerce Scenario
- FReTAL: Generalizing Deepfake Detection using Knowledge Distillation and Representation Learning
- Progressive and Aligned Pose Attention Transfer for Person Image Generation
- DeepFake MNIST+: A DeepFake Facial Animation Dataset
- JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion Retargeting
- ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis
- Flow Guided Transformable Bottleneck Networks for Motion Retargeting
- PriorityCut: Occlusion-guided Regularization for Warp-based Image Animation
- Video Reenactment as Inductive Bias for Content-Motion Disentanglement
- PIE: Portrait Image Embedding for Semantic Control
- DyStyle: Dynamic Neural Network for Multi-Attribute-Conditioned Style Editing
- Smile Like You Mean It: Driving Animatronic Robotic Face with Learned Models
- Unsupervised Landmark Detection Based Spatiotemporal Motion Estimation for 4D Dynamic Medical Images
- Impact of Benign Modifications on Discriminative Performance of Deepfake Detectors
- Dynamic Neural Garments
- AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person
- Cross-Identity Motion Transfer for Arbitrary Objects through Pose-Attentive Video Reassembling
- Grid Partitioned Attention: Efficient TransformerApproximation with Inductive Bias for High Resolution Detail Generation
- UniFaceGAN: A Unified Framework for Temporally Consistent Facial Video Editing
- 3D-Aware Semantic-Guided Generative Model for Human Synthesis
- Sparse to Dense Motion Transfer for Face Image Animation
- STALP: Style Transfer with Auxiliary Limited Pairing
- Intrinsic Temporal Regularization for High-resolution Human Video Synthesis
- Neural Re-Rendering of Humans from a Single Image
- LEGAN: Disentangled Manipulation of Directional Lighting and Facial Expressions by Leveraging Human Perceptual Judgements
- One-shot Face Reenactment Using Appearance Adaptive Normalization
- LI-Net: Large-Pose Identity-Preserving Face Reenactment Network