Action Unit Memory Network for Weakly Supervised Temporal Action Localization
arXiv:2104.14135
Abstract
Weakly supervised temporal action localization aims to detect and localize actions in untrimmed videos with only video-level labels during training. However, without frame-level annotations, it is challenging to achieve localization completeness and relieve background interference. In this paper, we present an Action Unit Memory Network (AUMN) for weakly supervised temporal action localization, which can mitigate the above two challenges by learning an action unit memory bank. In the proposed AUMN, two attention modules are designed to update the memory bank adaptively and learn action units specific classifiers. Furthermore, three effective mechanisms (diversity, homogeneity and sparsity) are designed to guide the updating of the memory network. To the best of our knowledge, this is the first work to explicitly model the action units with a memory network. Extensive experimental results on two standard benchmarks (THUMOS14 and ActivityNet) demonstrate that our AUMN performs favorably against state-of-the-art methods. Specifically, the average mAP of IoU thresholds from 0.1 to 0.5 on the THUMOS14 dataset is significantly improved from 47.0% to 52.1%.
arXiv commentAccepted by CVPR2021
References in corpus (10)
- Adam: A Method for Stochastic Optimization
- The THUMOS Challenge on Action Recognition for Videos "in the Wild"
- A Pursuit of Temporal Accuracy in General Activity Detection
- Temporal Localization of Fine-Grained Actions in Videos by Domain Transfer from Web Images
- Graph Convolutional Networks for Temporal Action Localization
- Step-by-step Erasion, One-by-one Collection: A Weakly Supervised Temporal Action Detector
- 3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization
- Weakly-Supervised Action Localization with Expectation-Maximization Multi-Instance Learning
- Segregated Temporal Assembly Recurrent Networks for Weakly Supervised Multiple Action Detection
- Adversarial Background-Aware Loss for Weakly-supervised Temporal Activity Localization