Knowing What, Where and When to Look: Efficient Video Action Modeling with Attention

Abstract

Attentive video modeling is essential for action recognition in unconstrained videos due to their rich yet redundant information over space and time. However, introducing attention in a deep neural network for action recognition is challenging for two reasons. First, an effective attention module needs to learn what (objects and their local motion patterns), where (spatially), and when (temporally) to focus on. Second, a video attention module must be efficient because existing action recognition models already suffer from high computational cost. To address both challenges, a novel What-Where-When (W3) video attention module is proposed. Departing from existing alternatives, our W3 module models all three facets of video attention jointly. Crucially, it is extremely efficient by factorizing the high-dimensional video feature data into low-dimensional meaningful spaces (1D channel vector for `what' and 2D spatial tensors for `where'), followed by lightweight temporal attention reasoning. Extensive experiments show that our attention model brings significant improvements to existing action recognition models, achieving new state-of-the-art performance on a number of benchmarks.

References in corpus (15)

• Distilling the Knowledge in a Neural Network
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Convolutional Sequence to Sequence Learning
• FitNets: Hints for Thin Deep Nets
• An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data
• Knowledge Distillation by On-the-Fly Native Ensemble
• Action Recognition using Visual Attention
• More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation
• Hierarchical Attention Network for Action Recognition in Videos
• On the effectiveness of task granularity for transfer learning
• Attention is All We Need: Nailing Down Object-centric Attention for Egocentric Activity Recognition
• Nonlocal Neural Networks, Nonlocal Diffusion and Nonlocal Modeling
• An Evaluation of Action Recognition Models on EPIC-Kitchens
• Action Classification and Highlighting in Videos

Cited by in corpus (6)

• Cross-Domain First Person Audio-Visual Action Recognition through Relative Norm Alignment
• SAFCAR: Structured Attention Fusion for Compositional Action Recognition
• Evaluating Transformers for Lightweight Action Recognition
• Egocentric Action Recognition by Video Attention and Temporal Context
• VideoLightFormer: Lightweight Action Recognition using Transformers
• Supervision Levels Scale (SLS)