Video Inpainting by Jointly Learning Temporal Structure and Spatial Details
arXiv:1806.08482
Abstract
We present a new data-driven video inpainting method for recovering missing regions of video frames. A novel deep learning architecture is proposed which contains two sub-networks: a temporal structure inference network and a spatial detail recovering network. The temporal structure inference network is built upon a 3D fully convolutional architecture: it only learns to complete a low-resolution video volume given the expensive computational cost of 3D convolution. The low resolution result provides temporal guidance to the spatial detail recovering network, which performs image-based inpainting with a 2D fully convolutional network to produce recovered video frames in their original resolution. Such two-step network design ensures both the spatial quality of each frame and the temporal coherence across frames. Our method jointly trains both sub-networks in an end-to-end manner. We provide qualitative and quantitative evaluation on three datasets, demonstrating that our method outperforms previous learning-based video inpainting methods.
arXiv commentAccepted by AAAI 2019
References in corpus (6)
- FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces
- Video Inpainting of Complex Scenes
- Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
- High-Resolution Shape Completion Using Deep Neural Networks for Global Structure and Local Geometry Inference
- Shape Inpainting using 3D Generative Adversarial Network and Recurrent Convolutional Networks
- Identity Preserving Face Completion for Large Ocular Region Occlusion
Cited by in corpus (6)
- Frame-Recurrent Video Inpainting by Robust Optical Flow Inference
- SiCloPe: Silhouette-Based Clothed People
- VORNet: Spatio-temporally Consistent Video Inpainting for Object Removal
- Semi-supervised Skin Detection by Network with Mutual Guidance
- AutoRemover: Automatic Object Removal for Autonomous Driving Videos
- Align-and-Attend Network for Globally and Locally Coherent Video Inpainting