Attentional Pooling for Action Recognition

Abstract

We introduce a simple yet surprisingly powerful model to incorporate attention in action recognition and human object interaction tasks. Our proposed attention module can be trained with or without extra supervision, and gives a sizable boost in accuracy while keeping the network size and computational cost nearly the same. It leads to significant improvements over state of the art base architecture on three standard action recognition benchmarks across still images and videos, and establishes new state of the art on MPII dataset with 12.5% relative improvement. We also perform an extensive analysis of our attention module both empirically and analytically. In terms of the latter, we introduce a novel derivation of bottom-up and top-down attention as low-rank approximations of bilinear pooling methods (typically used for fine-grained classification). From this perspective, our attention formulation suggests a novel characterization of action recognition as a fine-grained recognition problem.

In NIPS 2017. Project page: https://rohitgirdhar.github.io/AttentionalPoolingAction/

References in corpus (12)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• A simple neural network module for relational reasoning
• An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data
• Convolutional Two-Stream Network Fusion for Video Action Recognition
• Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
• Long-term Temporal Convolutions for Action Recognition
• Contextual Action Recognition with R*CNN
• Describing Common Human Visual Actions in Images
• Fine-grained Activity Recognition with Holistic and Pose based Features
• Joint Network based Attention for Action Recognition

Cited by in corpus (20)

• Attention Mechanisms in Computer Vision: A Survey
• Human Action Recognition from Various Data Modalities: A Review
• Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction
• Multi-Objective Matrix Normalization for Fine-grained Visual Recognition
• Second-Order Pooling for Graph Neural Networks
• Interaction-aware Spatio-temporal Pyramid Attention Networks for Action Classification
• Learning to Recognize Actions on Objects in Egocentric Video with Attention Dictionaries
• Learning Discriminative Motion Features Through Detection
• Weakly-Supervised Action Localization and Action Recognition using Global-Local Attention of 3D CNN
• Understanding Visual Ads by Aligning Symbols and Objects using Co-Attention
• GTNet:Guided Transformer Network for Detecting Human-Object Interactions
• This Reads Like That: Deep Learning for Interpretable Natural Language Processing
• Is Object Detection Necessary for Human-Object Interaction Recognition?
• Event and Activity Recognition in Video Surveillance for Cyber-Physical Systems
• Towards Scale-Invariant Graph-related Problem Solving by Iterative Homogeneous Graph Neural Networks
• Class Semantics-based Attention for Action Detection
• An Attention Model for group-level emotion recognition
• Loss Guided Activation for Action Recognition in Still Images
• Object-ABN: Learning to Generate Sharp Attention Maps for Action Recognition
• Efficient Modelling Across Time of Human Actions and Interactions