GTA: Global Temporal Attention for Video Action Understanding

Abstract

Self-attention learns pairwise interactions to model long-range dependencies, yielding great improvements for video action recognition. In this paper, we seek a deeper understanding of self-attention for temporal modeling in videos. We first demonstrate that the entangled modeling of spatio-temporal information by flattening all pixels is sub-optimal, failing to capture temporal relationships among frames explicitly. To this end, we introduce Global Temporal Attention (GTA), which performs global temporal attention on top of spatial attention in a decoupled manner. We apply GTA on both pixels and semantically similar regions to capture temporal relationships at different levels of spatial granularity. Unlike conventional self-attention that computes an instance-specific attention matrix, GTA directly learns a global attention matrix that is intended to encode temporal structures that generalize across different samples. We further augment GTA with a cross-channel multi-head fashion to exploit channel interactions for better temporal modeling. Extensive experiments on 2D and 3D networks demonstrate that our approach consistently enhances temporal modeling and provides state-of-the-art performance on three video action recognition datasets.

arXiv commentAccepted to BMVC 2021

References in corpus (15)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• The Kinetics Human Action Video Dataset
• Self-Attention Generative Adversarial Networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Attention-Based Models for Speech Recognition
• SGDR: Stochastic Gradient Descent with Warm Restarts
• An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data
• Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
• Non-local Neural Networks
• Context Encoding for Semantic Segmentation
• Compact Generalized Non-local Network
• More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation
• STM: SpatioTemporal and Motion Encoding for Action Recognition
• TEA: Temporal Excitation and Aggregation for Action Recognition
• Exploring Person Context and Local Scene Context for Object Detection

Cited by in corpus (3)

• Attention Mechanisms in Computer Vision: A Survey
• Feature Combination Meets Attention: Baidu Soccer Embeddings and Transformer based Temporal Detection
• Object-ABN: Learning to Generate Sharp Attention Maps for Action Recognition