Connectionist Temporal Modeling for Weakly Supervised Action Labeling

Abstract

We propose a weakly-supervised framework for action labeling in video, where only the order of occurring actions is required during training time. The key challenge is that the per-frame alignments between the input (video) and label (action) sequences are unknown during training. We address this by introducing the Extended Connectionist Temporal Classification (ECTC) framework to efficiently evaluate all possible alignments via dynamic programming and explicitly enforce their consistency with frame-to-frame visual similarities. This protects the model from distractions of visually inconsistent or degenerated alignments without the need of temporal supervision. We further extend our framework to the semi-supervised case when a few frames are sparsely annotated in a video. With less than 1% of labeled frames per video, our method is able to outperform existing semi-supervised approaches and achieve comparable performance to that of fully supervised approaches.

To appear in ECCV 2016

References in corpus (6)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Beyond Short Snippets: Deep Networks for Video Classification
• Sequence to Sequence -- Video to Text
• Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos
• Unsupervised Semantic Parsing of Video Collections
• Action Recognition by Hierarchical Mid-level Action Elements

Cited by in corpus (15)

• Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment
• TricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action Segmentation
• Weakly Supervised Action Localization by Sparse Temporal Pooling Network
• Forecasting future action sequences with attention: a new approach to weakly supervised action forecasting
• UntrimmedNets for Weakly Supervised Action Recognition and Detection
• Step-by-step Erasion, One-by-one Collection: A Weakly Supervised Temporal Action Detector
• Temporal Convolutional Networks for Action Segmentation and Detection
• Unsupervised Learning and Segmentation of Complex Activities from Video
• Learning to Localize and Align Fine-Grained Actions to Sparse Instructions
• Joint Discovery of Object States and Manipulation Actions
• WOAD: Weakly Supervised Online Action Detection in Untrimmed Videos
• Trespassing the Boundaries: Labeling Temporal Bounds for Object Interactions in Egocentric Video
• Spatio-Temporal Instance Learning: Action Tubes from Class Supervision
• Action Sets: Weakly Supervised Action Segmentation without Ordering Constraints
• Reconstructing and grounding narrated instructional videos in 3D