Spatial-Temporal Alignment Network for Action Recognition and Detection

Abstract

This paper studies how to introduce viewpoint-invariant feature representations that can help action recognition and detection. Although we have witnessed great progress of action recognition in the past decade, it remains challenging yet interesting how to efficiently model the geometric variations in large scale datasets. This paper proposes a novel Spatial-Temporal Alignment Network (STAN) that aims to learn geometric invariant representations for action recognition and action detection. The STAN model is very light-weighted and generic, which could be plugged into existing action recognition models like ResNet3D and the SlowFast with a very low extra computational cost. We test our STAN model extensively on AVA, Kinetics-400, AVA-Kinetics, Charades, and Charades-Ego datasets. The experimental results show that the STAN model can consistently improve the state of the arts in both action detection and action recognition tasks. We will release our data, models and code.

References in corpus (8)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• A Short Note on the Kinetics-700-2020 Human Action Dataset
• Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos
• The AVA-Kinetics Localized Human Actions Video Dataset
• Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization