Long Short-Term Transformer for Online Action Detection
arXiv:2107.03377
Abstract
We present Long Short-term TRansformer (LSTR), a temporal modeling algorithm for online action detection, which employs a long- and short-term memory mechanism to model prolonged sequence data. It consists of an LSTR encoder that dynamically leverages coarse-scale historical information from an extended temporal window (e.g., 2048 frames spanning of up to 8 minutes), together with an LSTR decoder that focuses on a short time window (e.g., 32 frames spanning 8 seconds) to model the fine-scale characteristics of the data. Compared to prior work, LSTR provides an effective and efficient method to model long videos with fewer heuristics, which is validated by extensive empirical analysis. LSTR achieves state-of-the-art performance on three standard online action detection benchmarks, THUMOS'14, TVSeries, and HACS Segment. Code has been made available at: https://xumingze0308.github.io/projects/lstr
arXiv commentNeurIPS 2021 Spotlight
References in corpus (21)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Is Space-Time Attention All You Need for Video Understanding?
- Linformer: Self-Attention with Linear Complexity
- The THUMOS Challenge on Action Recognition for Videos "in the Wild"
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
- Learnable pooling with Context Gating for video classification
- Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs
- Perceiver: General Perception with Iterative Attention
- BSN: Boundary Sensitive Network for Temporal Action Proposal Generation
- BMN: Boundary-Matching Network for Temporal Action Proposal Generation
- TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals
- CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos
- An Image is Worth 16x16 Words, What is a Video Worth?
- Activity Graph Transformer for Temporal Action Localization
- Temporally smooth online action detection using cycle-consistent future anticipation
- LAP-Net: Adaptive Features Sampling via Learning Action Progression for Online Action Detection
- StartNet: Online Detection of Action Start in Untrimmed Videos
- Privileged Knowledge Distillation for Online Action Detection
- VidTr: Video Transformer Without Convolutions