Learning Efficient Video Representation with Video Shuffle Networks
arXiv:1911.11319
Abstract
3D CNN shows its strong ability in learning spatiotemporal representation in recent video recognition tasks. However, inflating 2D convolution to 3D inevitably introduces additional computational costs, making it cumbersome in practical deployment. We consider whether there is a way to equip the conventional 2D convolution with temporal vision no requiring expanding its kernel. To this end, we propose the video shuffle, a parameter-free plug-in component that efficiently reallocates the inputs of 2D convolution so that its receptive field can be extended to the temporal dimension. In practical, video shuffle firstly divides each frame feature into multiple groups and then aggregate the grouped features via temporal shuffle operation. This allows the following 2D convolution aggregate the global spatiotemporal features. The proposed video shuffle can be flexibly inserted into popular 2D CNNs, forming the Video Shuffle Networks (VSN). With a simple yet efficient implementation, VSN performs surprisingly well on temporal modeling benchmarks. In experiments, VSN not only gains non-trivial improvements on Kinetics and Moments in Time, but also achieves state-of-the-art performance on Something-Something-V1, Something-Something-V2 datasets.
References in corpus (8)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Two-Stream Convolutional Networks for Action Recognition in Videos
- The Kinetics Human Action Video Dataset
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Learnable pooling with Context Gating for video classification
- SlowFast Networks for Video Recognition
- Temporal Modeling Approaches for Large-scale Youtube-8M Video Understanding
- Collaborative Spatio-temporal Feature Learning for Video Action Recognition