Spatio-Temporal Instance Learning: Action Tubes from Class Supervision

Abstract

The goal of this work is spatio-temporal action localization in videos, using only the supervision from video-level class labels. The state-of-the-art casts this weakly-supervised action localization regime as a Multiple Instance Learning problem, where instances are a priori computed spatio-temporal proposals. Rather than disconnecting the spatio-temporal learning from the training, we propose Spatio-Temporal Instance Learning, which enables action localization directly from box proposals in video frames. We outline the assumptions of our model and propose a max-margin objective and optimization with latent variables that enable spatio-temporal learning of actions from video labels. We also provide an efficient linking algorithm and two reranking strategies to facilitate and further improve the action localization. Experimental evaluation on four action datasets demonstrate the effectiveness of our approach for localization from weak supervision. Moreover, we show how to incorporate other supervision levels and mixtures, as a step towards determining optimal supervision strategies for action localization.

References in corpus (9)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos
• Weakly Supervised Action Localization by Sparse Temporal Pooling Network
• Weakly Supervised Action Labeling in Videos Under Ordering Constraints
• Finding Action Tubes
• Connectionist Temporal Modeling for Weakly Supervised Action Labeling
• Spatio-Temporal Action Detection with Cascade Proposal and Location Anticipation
• Localizing Actions from Video Labels and Pseudo-Annotations

Cited by in corpus (2)

• Weakly-Supervised Spatio-Temporal Anomaly Detection in Surveillance Video
• Discovering Multi-Label Actor-Action Association in a Weakly Supervised Setting