Skeleton-based Action Recognition via Spatial and Temporal Transformer Networks

Abstract

Skeleton-based Human Activity Recognition has achieved great interest in recent years as skeleton data has demonstrated being robust to illumination changes, body scales, dynamic camera views, and complex background. In particular, Spatial-Temporal Graph Convolutional Networks (ST-GCN) demonstrated to be effective in learning both spatial and temporal dependencies on non-Euclidean data such as skeleton graphs. Nevertheless, an effective encoding of the latent information underlying the 3D skeleton is still an open problem, especially when it comes to extracting effective information from joint motion patterns and their correlations. In this work, we propose a novel Spatial-Temporal Transformer network (ST-TR) which models dependencies between joints using the Transformer self-attention operator. In our ST-TR model, a Spatial Self-Attention module (SSA) is used to understand intra-frame interactions between different body parts, and a Temporal Self-Attention module (TSA) to model inter-frame correlations. The two are combined in a two-stream network, whose performance is evaluated on three large-scale datasets, NTU-RGB+D 60, NTU-RGB+D 120, and Kinetics Skeleton 400, consistently improving backbone results. Compared with methods that use the same input data, the proposed ST-TR achieves state-of-the-art performance on all datasets when using joints' coordinates as input, and results on-par with state-of-the-art when adding bones information.

Accepted at Computer Vision and Image Understanding (CVIU) 12 pages, 8 figures

References in corpus (10)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Two-Stream Convolutional Networks for Action Recognition in Videos
• The Kinetics Human Action Video Dataset
• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
• Deep Convolutional Networks on Graph-Structured Data
• An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data
• Point Transformer
• Parameter Efficient Multimodal Transformers for Video Representation Learning

Cited by in corpus (17)

• Transformers in Vision: A Survey
• Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognition
• Video Transformers: A Survey
• Continual Spatio-Temporal Graph Convolutional Networks
• D-STGCNT: A Dense Spatio-Temporal Graph Conv-GRU Network based on transformer for assessment of patient physical rehabilitation
• IIP-Transformer: Intra-Inter-Part Transformer for Skeleton-Based Action Recognition
• Action Capsules: Human Skeleton Action Recognition
• NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting
• STAR: Sparse Transformer-based Action Recognition
• SMART-Vision: Survey of Modern Action Recognition Techniques in Vision
• Hulk: A Universal Knowledge Translator for Human-Centric Tasks
• Wavelet-Decoupling Contrastive Enhancement Network for Fine-Grained Skeleton-Based Action Recognition
• Signal-SGN: A Spiking Graph Convolutional Network for Skeletal Action Recognition via Learning Temporal-Frequency Dynamics
• SkelVIT: Consensus of Vision Transformers for a Lightweight Skeleton-Based Action Recognition System
• Continual Inference: A Library for Efficient Online Inference with Deep Neural Networks in PyTorch
• Three-stream network for enriched Action Recognition
• Context-LGM: Leveraging Object-Context Relation for Context-Aware Object Recognition