A Survey on 3D Skeleton-Based Action Recognition Using Learning Method

Abstract

3D skeleton-based action recognition (3D SAR) has gained significant attention within the computer vision community, owing to the inherent advantages offered by skeleton data. As a result, a plethora of impressive works, including those based on conventional handcrafted features and learned feature extraction methods, have been conducted over the years. However, prior surveys on action recognition have primarily focused on video or RGB data-dominated approaches, with limited coverage of reviews related to skeleton data. Furthermore, despite the extensive application of deep learning methods in this field, there has been a notable absence of research that provides an introductory or comprehensive review from the perspective of deep learning architectures. To address these limitations, this survey first underscores the importance of action recognition and emphasizes the significance of 3D skeleton data as a valuable modality. Subsequently, we provide a comprehensive introduction to mainstream action recognition techniques based on four fundamental deep architectures, i.e., Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), Graph Convolutional Network (GCN), and Transformers. All methods with the corresponding architectures are then presented in a data-driven manner with detailed discussion. Finally, we offer insights into the current largest 3D skeleton dataset, NTU-RGB+D, and its new edition, NTU-RGB+D 120, along with an overview of several top-performing algorithms on these datasets. To the best of our knowledge, this research represents the first comprehensive discussion of deep learning-based action recognition using 3D skeleton data.

Accepted by Cyborg and Bionic Systems (CBS), 11 pages, 6 figures

References in corpus (14)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
• OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
• Quasi-Recurrent Neural Networks
• SkeleMotion: A New Representation of Skeleton Joint Sequences Based on Motion Information for 3D Action Recognition
• SlowFast Networks for Video Recognition
• Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN
• Two-Stream 3D Convolutional Neural Network for Skeleton-Based Action Recognition
• Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation
• Make Skeleton-based Action Recognition Model Smaller, Faster and Better
• Memory Attention Networks for Skeleton-based Action Recognition
• Adding Attentiveness to the Neurons in Recurrent Neural Networks
• Kinematic-Layout-aware Random Forests for Depth-based Action Recognition

Cited by in corpus (8)

• Human Action Recognition from Various Data Modalities: A Review
• Skeleton-based Action Recognition via Spatial and Temporal Transformer Networks
• Vision-Based Human Pose Estimation via Deep Learning: A Survey
• Logsig-RNN: a novel network for robust and efficient skeleton-based action recognition
• BASAR:Black-box Attack on Skeletal Action Recognition
• Graph Convolution with Low-rank Learnable Local Filters
• NTU-X: An Enhanced Large-scale Dataset for Improving Pose-based Recognition of Subtle Human Actions
• GeomNet: A Neural Network Based on Riemannian Geometries of SPD Matrix Space and Cholesky Space for 3D Skeleton-Based Interaction Recognition