Action Classification and Highlighting in Videos

Abstract

Inspired by recent advances in neural machine translation, that jointly align and translate using encoder-decoder networks equipped with attention, we propose an attentionbased LSTM model for human activity recognition. Our model jointly learns to classify actions and highlight frames associated with the action, by attending to salient visual information through a jointly learned soft-attention networks. We explore attention informed by various forms of visual semantic features, including those encoding actions, objects and scenes. We qualitatively show that soft-attention can learn to effectively attend to important objects and scene information correlated with specific human actions. Further, we show that, quantitatively, our attention-based LSTM outperforms the vanilla LSTM and CNN models used by stateof-the-art methods. On a large-scale youtube video dataset, ActivityNet, our model outperforms competing methods in action classification.

References in corpus (10)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Theano: new features and speed improvements
• Action Recognition using Visual Attention
• Sequence to Sequence -- Video to Text
• Describing Videos by Exploiting Temporal Structure
• Blocks and Fuel: Frameworks for deep learning
• Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos

Cited by in corpus (2)

• Knowing What, Where and When to Look: Efficient Video Action Modeling with Attention
• Interpretable Spatio-temporal Attention for Video Action Recognition