MotionSqueeze: Neural Motion Feature Learning for Video Understanding
arXiv:2007.09933
Abstract
Motion plays a crucial role in understanding videos and most state-of-the-art neural models for video classification incorporate motion information typically using optical flows extracted by a separate off-the-shelf method. As the frame-by-frame optical flows require heavy computation, incorporating motion information has remained a major computational bottleneck for video understanding. In this work, we replace external and heavy computation of optical flows with internal and light-weight learning of motion features. We propose a trainable neural module, dubbed MotionSqueeze, for effective motion feature extraction. Inserted in the middle of any neural network, it learns to establish correspondences across frames and convert them into motion features, which are readily fed to the next downstream layer for better prediction. We demonstrate that the proposed method provides a significant gain on four standard benchmarks for action recognition with only a small amount of additional cost, outperforming the state of the art on Something-Something-V1&V2 datasets.
arXiv commentAccepted to ECCV 2020
References in corpus (15)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Deep Residual Learning for Image Recognition
- The Kinetics Human Action Video Dataset
- MobileNetV2: Inverted Residuals and Linear Bottlenecks
- Convolutional Two-Stream Network Fusion for Video Action Recognition
- Xception: Deep Learning with Depthwise Separable Convolutions
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
- A Closer Look at Spatiotemporal Convolutions for Action Recognition
- Learning Spatiotemporal Features with 3D Convolutional Networks
- Non-local Neural Networks
- STM: SpatioTemporal and Motion Encoding for Action Recognition
- Convolutional neural network architecture for geometric matching
- End-to-End Learning of Motion Representation for Video Understanding
- SCNet: Learning Semantic Correspondence
- Learning Video Representations from Correspondence Proposals