HeadOn: Real-time Reenactment of Human Portrait Videos
arXiv:1805.11729 · doi:10.1145/3197517.3201350
Abstract
We propose HeadOn, the first real-time source-to-target reenactment approach for complete human portrait videos that enables transfer of torso and head motion, face expression, and eye gaze. Given a short RGB-D video of the target actor, we automatically construct a personalized geometry proxy that embeds a parametric head, eye, and kinematic torso model. A novel real-time reenactment algorithm employs this proxy to photo-realistically map the captured motion from the source actor to the target actor. On top of the coarse geometric proxy, we propose a video-based rendering technique that composites the modified target portrait video via view- and pose-dependent texturing, and creates photo-realistic imagery of the target actor under novel torso and head poses, facial expressions, and gaze directions. To this end, we propose a robust tracking of the face and torso of the source actor. We extensively evaluate our approach and show significant improvements in enabling much greater flexibility in creating realistic reenacted output videos.
arXiv commentVideo: https://www.youtube.com/watch?v=7Dg49wv2c_g Presented at Siggraph'18
References in corpus (3)
Cited by in corpus (10)
- The Creation and Detection of Deepfakes: A Survey
- Head2Head++: Deep Facial Attributes Re-Targeting
- Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
- Forbidden knowledge in machine learning -- Reflections on the limits of research and publication
- Protecting Against Image Translation Deepfakes by Leaking Universal Perturbations from Black-Box Neural Networks
- Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation
- EgoFace: Egocentric Face Performance Capture and Videorealistic Reenactment
- FReeNet: Multi-Identity Face Reenactment
- Metric Learning for Anti-Compression Facial Forgery Detection
- Sparse to Dense Motion Transfer for Face Image Animation