VideoLSTM Convolves, Attends and Flows for Action Recognition

Abstract

We present a new architecture for end-to-end sequence learning of actions in video, we call VideoLSTM. Rather than adapting the video to the peculiarities of established recurrent or convolutional architectures, we adapt the architecture to fit the requirements of the video medium. Starting from the soft-Attention LSTM, VideoLSTM makes three novel contributions. First, video has a spatial layout. To exploit the spatial correlation we hardwire convolutions in the soft-Attention LSTM architecture. Second, motion not only informs us about the action content, but also guides better the attention towards the relevant spatio-temporal locations. We introduce motion-based attention. And finally, we demonstrate how the attention from VideoLSTM can be used for action localization by relying on just the action class label. Experiments and comparisons on challenging datasets for action classification and localization support our claims.

References in corpus (7)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Deep Residual Learning for Image Recognition
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Unsupervised Learning of Video Representations using LSTMs
• Bag of Visual Words and Fusion Methods for Action Recognition: Comprehensive Study and Good Practice
• Learning to track for spatio-temporal action localization

Cited by in corpus (17)

• Attention Mechanisms in Computer Vision: A Survey
• Flow-Guided Feature Aggregation for Video Object Detection
• Interaction-aware Spatio-temporal Pyramid Attention Networks for Action Classification
• Towards High Performance Video Object Detection for Mobiles
• Spatially Adaptive Computation Time for Residual Networks
• Neural Person Search Machines
• Semantic Video Segmentation by Gated Recurrent Flow Propagation
• HCMS: Hierarchical and Conditional Modality Selection for Efficient Video Recognition
• Self-Supervised Video Representation Learning With Odd-One-Out Networks
• Learning to score the figure skating sports videos
• Hierarchical Multi-scale Attention Networks for Action Recognition
• Video Representation Learning and Latent Concept Mining for Large-scale Multi-label Video Classification
• Cross-Modal Message Passing for Two-stream Fusion
• Temporal-Spatial Mapping for Action Recognition
• AdaScan: Adaptive Scan Pooling in Deep Convolutional Neural Networks for Human Action Recognition in Videos
• Efficient Modelling Across Time of Human Actions and Interactions
• Making a Case for Learning Motion Representations with Phase