Weakly Supervised Action Selection Learning in Video
arXiv:2105.02439
Abstract
Localizing actions in video is a core task in computer vision. The weakly supervised temporal localization problem investigates whether this task can be adequately solved with only video-level labels, significantly reducing the amount of expensive and error-prone annotation that is required. A common approach is to train a frame-level classifier where frames with the highest class probability are selected to make a video-level prediction. Frame level activations are then used for localization. However, the absence of frame-level annotations cause the classifier to impart class bias on every frame. To address this, we propose the Action Selection Learning (ASL) approach to capture the general concept of action, a property we refer to as "actionness". Under ASL, the model is trained with a novel class-agnostic task to predict which frames will be selected by the classifier. Empirically, we show that ASL outperforms leading baselines on two popular benchmarks THUMOS-14 and ActivityNet-1.2, with 10.3% and 5.7% relative improvement respectively. We further analyze the properties of ASL and demonstrate the importance of actionness. Full code for this work is available here: https://github.com/layer6ai-labs/ASL.
arXiv commentCVPR 2021
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- The Kinetics Human Action Video Dataset
- Multiple Instance Learning: A Survey of Problem Characteristics and Applications
- BSN: Boundary Sensitive Network for Temporal Action Proposal Generation
- BMN: Boundary-Matching Network for Temporal Action Proposal Generation
- Marginalized Average Attentional Network for Weakly-Supervised Learning
- 3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization
- Weakly-Supervised Action Localization with Expectation-Maximization Multi-Instance Learning
- Actionness Estimation Using Hybrid Fully Convolutional Networks
- Weakly-Supervised Action Localization by Generative Attention Modeling
- Searching Action Proposals via Spatial Actionness Estimation and Temporal Path Inference and Tracking