STM: SpatioTemporal and Motion Encoding for Action Recognition
arXiv:1908.02486
Abstract
Spatiotemporal and motion features are two complementary and crucial information for video action recognition. Recent state-of-the-art methods adopt a 3D CNN stream to learn spatiotemporal features and another flow stream to learn motion features. In this work, we aim to efficiently encode these two features in a unified 2D framework. To this end, we first propose an STM block, which contains a Channel-wise SpatioTemporal Module (CSTM) to present the spatiotemporal features and a Channel-wise Motion Module (CMM) to efficiently encode motion features. We then replace original residual blocks in the ResNet architecture with STM blcoks to form a simple yet effective STM network by introducing very limited extra computation cost. Extensive experiments demonstrate that the proposed STM network outperforms the state-of-the-art methods on both temporal-related datasets (i.e., Something-Something v1 & v2 and Jester) and scene-related datasets (i.e., Kinetics-400, UCF-101, and HMDB-51) with the help of encoding spatiotemporal and motion features together.
Accepted by ICCV2019
References in corpus (6)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Two-Stream Convolutional Networks for Action Recognition in Videos
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Spatiotemporal Residual Networks for Video Action Recognition
- SlowFast Networks for Video Recognition
- StNet: Local and Global Spatial-Temporal Modeling for Action Recognition
Cited by in corpus (11)
- Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers
- TAM: Temporal Adaptive Module for Video Recognition
- GTA: Global Temporal Attention for Video Action Understanding
- Hierarchical Contrastive Motion Learning for Video Action Recognition
- Attention Distillation for Learning Video Representations
- A Multigrid Method for Efficiently Training Video Models
- MotionSqueeze: Neural Motion Feature Learning for Video Understanding
- TEAM-Net: Multi-modal Learning for Video Action Recognition with Partial Decoding
- LIGAR: Lightweight General-purpose Action Recognition
- Perceptron Synthesis Network: Rethinking the Action Scale Variances in Videos
- Efficient Modelling Across Time of Human Actions and Interactions