Deep Face Video Inpainting via UV Mapping
arXiv:2109.00681 · doi:10.1109/TIP.2023.3240835
Abstract
This paper addresses the problem of face video inpainting. Existing video inpainting methods target primarily at natural scenes with repetitive patterns. They do not make use of any prior knowledge of the face to help retrieve correspondences for the corrupted face. They therefore only achieve sub-optimal results, particularly for faces under large pose and expression variations where face components appear very differently across frames. In this paper, we propose a two-stage deep learning method for face video inpainting. We employ 3DMM as our 3D face prior to transform a face between the image space and the UV (texture) space. In Stage I, we perform face inpainting in the UV space. This helps to largely remove the influence of face poses and expressions and makes the learning task much easier with well aligned face features. We introduce a frame-wise attention module to fully exploit correspondences in neighboring frames to assist the inpainting task. In Stage II, we transform the inpainted face regions back to the image space and perform face video refinement that inpaints any background regions not covered in Stage I and also refines the inpainted face regions. Extensive experiments have been carried out which show our method can significantly outperform methods based merely on 2D information, especially for faces under large pose and expression variations. Project page: https://ywq.github.io/FVIP
TIP 2023
References in corpus (17)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)
- EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
- Video Inpainting of Complex Scenes
- Large Scale Image Completion via Co-Modulated Generative Adversarial Networks
- SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting
- Learnable Gated Temporal Shift Module for Deep Video Inpainting
- OSTeC: One-Shot Texture Completion
- Deep Portrait Image Completion and Extrapolation
- FaceShop: Deep Sketch-based Face Image Editing
- LaFIn: Generative Landmark Guided Face Inpainting
- Frame-Recurrent Video Inpainting by Robust Optical Flow Inference
- Improved Techniques for GAN based Facial Inpainting
- Domain Embedded Multi-model Generative Adversarial Networks for Image-based Face Inpainting
- Task-agnostic Temporally Consistent Facial Video Editing
- Image Inpainting with Edge-guided Learnable Bidirectional Attention Maps
- NÜWA-LIP: Language Guided Image Inpainting with Defect-free VQGAN