Weakly Supervised Action Localization by Sparse Temporal Pooling Network

Abstract

We propose a weakly supervised temporal action localization algorithm on untrimmed videos using convolutional neural networks. Our algorithm learns from video-level class labels and predicts temporal intervals of human actions with no requirement of temporal localization annotations. We design our network to identify a sparse subset of key segments associated with target actions in a video using an attention module and fuse the key segments through adaptive temporal pooling. Our loss function is comprised of two terms that minimize the video-level action classification error and enforce the sparsity of the segment selection. At inference time, we extract and score temporal proposals using temporal class activations and class-agnostic attentions to estimate the time intervals that correspond to target actions. The proposed algorithm attains state-of-the-art results on the THUMOS14 dataset and outstanding performance on ActivityNet1.3 even with its weak supervision.

Accepted to CVPR 2018

References in corpus (18)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Spatiotemporal Residual Networks for Video Action Recognition
• Convolutional Two-Stream Network Fusion for Video Action Recognition
• Learning Deep Features for Discriminative Localization
• Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs
• Learning Spatiotemporal Features with 3D Convolutional Networks
• A Pursuit of Temporal Accuracy in General Activity Detection
• R-C3D: Region Convolutional 3D Network for Temporal Activity Detection
• Untrimmed Video Classification for Activity Detection: submission to ActivityNet Challenge
• Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
• Weakly Supervised Action Labeling in Videos Under Ordering Constraints
• Finding Action Tubes
• Connectionist Temporal Modeling for Weakly Supervised Action Labeling
• Actionness Estimation Using Hybrid Fully Convolutional Networks
• AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions
• Action Search: Spotting Actions in Videos and Its Application to Temporal Action Localization

Cited by in corpus (15)

• VideoMix: Rethinking Data Augmentation for Video Classification
• Weakly-Supervised Action Localization with Expectation-Maximization Multi-Instance Learning
• Unsupervised Two-Stage Anomaly Detection
• Weak Supervision and Referring Attention for Temporal-Textual Association Learning
• Point-Level Temporal Action Localization: Bridging Fully-supervised Proposals to Weakly-supervised Losses
• Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing
• Condensation-Net: Memory-Efficient Network Architecture with Cross-Channel Pooling Layers and Virtual Feature Maps
• Multi-shot Temporal Event Localization: a Benchmark
• Spatio-Temporal Instance Learning: Action Tubes from Class Supervision
• Cross-Class Relevance Learning for Temporal Concept Localization
• Weakly-Supervised Temporal Action Localization Through Local-Global Background Modeling
• Human Action Sequence Classification
• Class Semantics-based Attention for Action Detection
• CoLA: Weakly-Supervised Temporal Action Localization with Snippet Contrastive Learning
• The Blessings of Unlabeled Background in Untrimmed Videos