Spatial Temporal Transformer Network for Skeleton-based Action Recognition

Abstract

Skeleton-based human action recognition has achieved a great interest in recent years, as skeleton data has been demonstrated to be robust to illumination changes, body scales, dynamic camera views, and complex background. Nevertheless, an effective encoding of the latent information underlying the 3D skeleton is still an open problem. In this work, we propose a novel Spatial-Temporal Transformer network (ST-TR) which models dependencies between joints using the Transformer self-attention operator. In our ST-TR model, a Spatial Self-Attention module (SSA) is used to understand intra-frame interactions between different body parts, and a Temporal Self-Attention module (TSA) to model inter-frame correlations. The two are combined in a two-stream network which outperforms state-of-the-art models using the same input data on both NTU-RGB+D 60 and NTU-RGB+D 120.

arXiv commentAccepted as ICPRW2020 (FBE2020, Workshop on Facial and Body Expressions, micro-expressions and behavior recognition) 8 pages, 2 figures. arXiv admin note: substantial text overlap with arXiv:2008.07404

References in corpus (5)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
• Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
• Skeleton-Based Human Action Recognition with Global Context-Aware Attention LSTM Networks
• DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks

Cited by in corpus (11)

• Transformers in Vision: A Survey
• Human Action Recognition from Various Data Modalities: A Review
• NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting
• GaitFormer: Learning Gait Representations with Noisy Multi-Task Learning
• STAR: Sparse Transformer-based Action Recognition
• SMART-Vision: Survey of Modern Action Recognition Techniques in Vision
• Video-based Contrastive Learning on Decision Trees: from Action Recognition to Autism Diagnosis
• Action Recognition in Real-World Ambient Assisted Living Environment
• On the Relevance of Temporal Features for Medical Ultrasound Video Recognition
• Zero-Shot Skeleton-Based Action Recognition With Prototype-Guided Feature Alignment
• Three-stream network for enriched Action Recognition