Sparse to Dense Motion Transfer for Face Image Animation

Abstract

Face image animation from a single image has achieved remarkable progress. However, it remains challenging when only sparse landmarks are available as the driving signal. Given a source face image and a sequence of sparse face landmarks, our goal is to generate a video of the face imitating the motion of landmarks. We develop an efficient and effective method for motion transfer from sparse landmarks to the face image. We then combine global and local motion estimation in a unified model to faithfully transfer the motion. The model can learn to segment the moving foreground from the background and generate not only global motion, such as rotation and translation of the face, but also subtle local motion such as the gaze change. We further improve face landmark detection on videos. With temporally better aligned landmark sequences for training, our method can generate temporally coherent videos with higher visual quality. Experiments suggest we achieve results comparable to the state-of-the-art image driven method on the same identity testing and better results on cross identity testing.

Accepted by ICCV 2021 Advances in Image Manipulation Workshop

References in corpus (7)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network
• Deferred Neural Rendering: Image Synthesis using Neural Textures
• Supervision by Registration and Triangulation for Landmark Detection
• Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss
• Zooming SlowMo: An Efficient One-Stage Framework for Space-Time Video Super-Resolution
• FaceController: Controllable Attribute Editing for Face in the Wild