A Temporally-Aware Interpolation Network for Video Frame Inpainting
arXiv:1803.07218
Abstract
We propose the first deep learning solution to video frame inpainting, a challenging instance of the general video inpainting problem with applications in video editing, manipulation, and forensics. Our task is less ambiguous than frame interpolation and video prediction because we have access to both the temporal context and a partial glimpse of the future, allowing us to better evaluate the quality of a model's predictions objectively. We devise a pipeline composed of two modules: a bidirectional video prediction module, and a temporally-aware frame interpolation module. The prediction module makes two intermediate predictions of the missing frames, one conditioned on the preceding frames and the other conditioned on the following frames, using a shared convolutional LSTM-based encoder-decoder. The interpolation module blends the intermediate predictions to form the final result. Specifically, it utilizes time information and hidden activations from the video prediction module to resolve disagreements between the predictions. Our experiments demonstrate that our approach produces more accurate and qualitatively satisfying results than a state-of-the-art video prediction method and many strong frame inpainting baselines.
References in corpus (12)
- Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Spectral Normalization for Generative Adversarial Networks
- Decomposing Motion and Content for Natural Video Sequence Prediction
- Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
- Video (language) modeling: a baseline for generative models of natural videos
- Video Inpainting of Complex Scenes
- Video Frame Interpolation via Adaptive Separable Convolution
- Video Frame Interpolation via Adaptive Convolution
- Video Frame Synthesis using Deep Voxel Flow
- Deep Stacked Networks with Residual Polishing for Image Inpainting
- Improving Consistency and Correctness of Sequence Inpainting using Semantically Guided Generative Adversarial Network