Learnable Gated Temporal Shift Module for Deep Video Inpainting
arXiv:1907.01131
Abstract
How to efficiently utilize temporal information to recover videos in a consistent way is the main issue for video inpainting problems. Conventional 2D CNNs have achieved good performance on image inpainting but often lead to temporally inconsistent results where frames will flicker when applied to videos (see https://www.youtube.com/watch?v=87Vh1HDBjD0&list=PLPoVtv-xp_dL5uckIzz1PKwNjg1yI0I94&index=1); 3D CNNs can capture temporal information but are computationally intensive and hard to train. In this paper, we present a novel component termed Learnable Gated Temporal Shift Module (LGTSM) for video inpainting models that could effectively tackle arbitrary video masks without additional parameters from 3D convolutions. LGTSM is designed to let 2D convolutions make use of neighboring frames more efficiently, which is crucial for video inpainting. Specifically, in each layer, LGTSM learns to shift some channels to its temporal neighbors so that 2D convolutions could be enhanced to handle temporal information. Meanwhile, a gated convolution is applied to the layer to identify the masked areas that are poisoning for conventional convolutions. On the FaceForensics and Free-form Video Inpainting (FVI) dataset, our model achieves state-of-the-art results with simply 33% of parameters and inference time.
Accepted to BMVC 2019
References in corpus (11)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Two-Stream Convolutional Networks for Action Recognition in Videos
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Spectral Normalization for Generative Adversarial Networks
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
- EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
- FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces
- Generative Image Inpainting with Contextual Attention
- Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN
- VORNet: Spatio-temporally Consistent Video Inpainting for Object Removal
- Temporal Bilinear Networks for Video Action Recognition
Cited by in corpus (12)
- Free-form tumor synthesis in computed tomography images via richer generative adversarial network
- Decoupled Spatial-Temporal Transformer for Video Inpainting
- Deep Long Audio Inpainting
- Deep Face Video Inpainting via UV Mapping
- Learning Joint Spatial-Temporal Transformations for Video Inpainting
- DeViT: Deformed Vision Transformers in Video Inpainting
- FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting
- Headset: Human emotion awareness under partial occlusions multimodal dataset
- Expression-aware video inpainting for HMD removal in XR applications
- Progressive Temporal Feature Alignment Network for Video Inpainting
- Deep Video Inpainting Guided by Audio-Visual Self-Supervision
- Depth Guided Adaptive Meta-Fusion Network for Few-shot Video Recognition