End-to-end Learning of Action Detection from Frame Glimpses in Videos

Abstract

In this work we introduce a fully end-to-end approach for action detection in videos that learns to directly predict the temporal bounds of actions. Our intuition is that the process of detecting actions is naturally one of observation and refinement: observing moments in video, and refining hypotheses about when an action is occurring. Based on this insight, we formulate our model as a recurrent neural network-based agent that interacts with a video over time. The agent observes video frames and decides both where to look next and when to emit a prediction. Since backpropagation is not adequate in this non-differentiable setting, we use REINFORCE to learn the agent's decision policy. Our model achieves state-of-the-art results on the THUMOS'14 and ActivityNet datasets while observing only a fraction (2% or less) of the video frames.

Update to version in CVPR 2016 proceedings

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Recurrent Models of Visual Attention
• Multiple Object Recognition with Visual Attention
• Beyond Short Snippets: Deep Networks for Video Classification
• Attention for Fine-Grained Categorization
• Reinforcement Learning Neural Turing Machines - Revised
• Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos
• Exploiting Image-trained CNN Architectures for Unconstrained Video Classification
• Learning to track for spatio-temporal action localization
• Finding Action Tubes

Cited by in corpus (9)

• Hierarchical Object Detection with Deep Reinforcement Learning
• Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
• Temporal Dynamic Graph LSTM for Action-driven Video Object Detection
• LAP-Net: Adaptive Features Sampling via Learning Action Progression for Online Action Detection
• Asynchronous Temporal Fields for Action Recognition
• Online Action Detection
• Cricket stroke extraction: Towards creation of a large-scale cricket actions dataset
• Rethinking Online Action Detection in Untrimmed Videos: A Novel Online Evaluation Protocol
• Reinforced Attention for Few-Shot Learning and Beyond