Skeleton-Based Human Action Recognition with Global Context-Aware Attention LSTM Networks

Abstract

Human action recognition in 3D skeleton sequences has attracted a lot of research attention. Recently, Long Short-Term Memory (LSTM) networks have shown promising performance in this task due to their strengths in modeling the dependencies and dynamics in sequential data. As not all skeletal joints are informative for action recognition, and the irrelevant joints often bring noise which can degrade the performance, we need to pay more attention to the informative ones. However, the original LSTM network does not have explicit attention ability. In this paper, we propose a new class of LSTM network, Global Context-Aware Attention LSTM (GCA-LSTM), for skeleton based action recognition. This network is capable of selectively focusing on the informative joints in each frame of each skeleton sequence by using a global context memory cell. To further improve the attention capability of our network, we also introduce a recurrent attention mechanism, with which the attention performance of the network can be enhanced progressively. Moreover, we propose a stepwise training scheme in order to train our network effectively. Our approach achieves state-of-the-art performance on five challenging benchmark datasets for skeleton based action recognition.

Cited by in corpus (13)

• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
• Human Action Recognition from Various Data Modalities: A Review
• Skeleton-based Action Recognition via Spatial and Temporal Transformer Networks
• Richly Activated Graph Convolutional Network for Robust Skeleton-based Action Recognition
• Spatial Temporal Transformer Network for Skeleton-based Action Recognition
• Human in Events: A Large-Scale Benchmark for Human-centric Video Analysis in Complex Events
• Two-person Graph Convolutional Network for Skeleton-based Human Interaction Recognition
• Interactive Spatiotemporal Token Attention Network for Skeleton-based General Interactive Action Recognition
• SpATr: MoCap 3D Human Action Recognition based on Spiral Auto-encoder and Transformer Network
• Real-Time Hand Gesture Recognition: Integrating Skeleton-Based Data Fusion and Multi-Stream CNN
• MK-SGN: A Spiking Graph Convolutional Network with Multimodal Fusion and Knowledge Distillation for Skeleton-based Action Recognition
• SkeletonX: Data-Efficient Skeleton-based Action Recognition via Cross-sample Feature Aggregation
• De-biasing Skeleton-based Action Recognition with Convex Hull Adaptive Shift