V4D:4D Convolutional Neural Networks for Video-level Representation Learning
arXiv:2002.07442
Abstract
Most existing 3D CNNs for video representation learning are clip-based methods, and thus do not consider video-level temporal evolution of spatio-temporal features. In this paper, we propose Video-level 4D Convolutional Neural Networks, referred as V4D, to model the evolution of long-range spatio-temporal representation with 4D convolutions, and at the same time, to preserve strong 3D spatio-temporal representation with residual connections. Specifically, we design a new 4D residual block able to capture inter-clip interactions, which could enhance the representation power of the original clip-level 3D CNNs. The 4D residual blocks can be easily integrated into the existing 3D CNNs to perform long-range modeling hierarchically. We further introduce the training and inference methods for the proposed V4D. Extensive experiments are conducted on three video recognition benchmarks, where V4D achieves excellent results, surpassing recent 3D CNNs by a large margin.
To appear in ICLR2020
References in corpus (7)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Two-Stream Convolutional Networks for Action Recognition in Videos
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
- SlowFast Networks for Video Recognition
- Real-time Action Recognition with Enhanced Motion Vector CNNs
- Learning Video Representations from Correspondence Proposals
- Action4D: Real-time Action Recognition in the Crowd and Clutter
Cited by in corpus (5)
- Human Action Recognition from Various Data Modalities: A Review
- Context-Aware RCNN: A Baseline for Action Detection in Videos
- Recent Progress in Appearance-based Action Recognition
- Multiple Visual-Semantic Embedding for Video Retrieval from Query Sentence
- Symmetric Dilated Convolution for Surgical Gesture Recognition