Learnable Sampling 3D Convolution for Video Enhancement and Action Recognition
arXiv:2011.10974
Abstract
A key challenge in video enhancement and action recognition is to fuse useful information from neighboring frames. Recent works suggest establishing accurate correspondences between neighboring frames before fusing temporal information. However, the generated results heavily depend on the quality of correspondence estimation. In this paper, we propose a more robust solution: \emph{sampling and fusing multi-level features} across neighborhood frames to generate the results. Based on this idea, we introduce a new module to improve the capability of 3D convolution, namely, learnable sampling 3D convolution (\emph{LS3D-Conv}). We add learnable 2D offsets to 3D convolution which aims to sample locations on spatial feature maps across frames. The offsets can be learned for specific tasks. The \emph{LS3D-Conv} can flexibly replace 3D convolution layers in existing 3D networks and get new architectures, which learns the sampling at multiple feature levels. The experiments on video interpolation, video super-resolution, video denoising, and action recognition demonstrate the effectiveness of our approach.
References in corpus (9)
- The Kinetics Human Action Video Dataset
- Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
- Video-to-Video Synthesis
- Deformable Non-local Network for Video Super-Resolution
- TDAN: Temporally Deformable Alignment Network for Video Super-Resolution
- 3DSRnet: Video Super-resolution using 3D Convolutional Neural Networks
- Learning Deformable Kernels for Image and Video Denoising
- Hierarchical Video Generation from Orthogonal Information: Optical Flow and Texture
- Deep SR-ITM: Joint Learning of Super-Resolution and Inverse Tone-Mapping for 4K UHD HDR Applications