Segregated Temporal Assembly Recurrent Networks for Weakly Supervised Multiple Action Detection
arXiv:1811.07460
Abstract
This paper proposes a segregated temporal assembly recurrent (STAR) network for weakly-supervised multiple action detection. The model learns from untrimmed videos with only supervision of video-level labels and makes prediction of intervals of multiple actions. Specifically, we first assemble video clips according to class labels by an attention mechanism that learns class-variable attention weights and thus helps the noise relieving from background or other actions. Secondly, we build temporal relationship between actions by feeding the assembled features into an enhanced recurrent neural network. Finally, we transform the output of recurrent neural network into the corresponding action distribution. In order to generate more precise temporal proposals, we design a score term called segregated temporal gradient-weighted class activation mapping (ST-GradCAM) fused with attention weights. Experiments on THUMOS'14 and ActivityNet1.3 datasets show that our approach outperforms the state-of-the-art weakly-supervised method, and performs at par with the fully-supervised counterparts.
arXiv commentAccepted to Proc. AAAI Conference on Artificial Intelligence 2019
References in corpus (17)
- Two-Stream Convolutional Networks for Action Recognition in Videos
- The Kinetics Human Action Video Dataset
- STC: A Simple to Complex Framework for Weakly-supervised Semantic Segmentation
- Multi-Instance Multi-Label Learning
- Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs
- Modeling Coverage for Neural Machine Translation
- Temporal Action Detection with Structured Segment Networks
- A Pursuit of Temporal Accuracy in General Activity Detection
- R-C3D: Region Convolutional 3D Network for Temporal Activity Detection
- Untrimmed Video Classification for Activity Detection: submission to ActivityNet Challenge
- BSN: Boundary Sensitive Network for Temporal Action Proposal Generation
- Temporal Context Network for Activity Localization in Videos
- CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos
- Cascaded Boundary Regression for Temporal Action Detection
- Exploring Temporal Preservation Networks for Precise Temporal Action Localization
- UntrimmedNets for Weakly Supervised Action Recognition and Detection
- Learning to Separate Object Sounds by Watching Unlabeled Video
Cited by in corpus (4)
- Action Unit Memory Network for Weakly Supervised Temporal Action Localization
- Towards Train-Test Consistency for Semi-supervised Temporal Action Localization
- Scale Matters: Temporal Scale Aggregation Network for Precise Action Localization in Untrimmed Videos
- Foreground-Action Consistency Network for Weakly Supervised Temporal Action Localization