NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis

Abstract

Recent approaches in depth-based human activity analysis achieved outstanding performance and proved the effectiveness of 3D representation for classification of action classes. Currently available depth-based and RGB+D-based action recognition benchmarks have a number of limitations, including the lack of training samples, distinct class labels, camera views and variety of subjects. In this paper we introduce a large-scale dataset for RGB+D human action recognition with more than 56 thousand video samples and 4 million frames, collected from 40 distinct subjects. Our dataset contains 60 different action classes including daily, mutual, and health-related actions. In addition, we propose a new recurrent neural network structure to model the long-term temporal correlation of the features for each body part, and utilize them for better action classification. Experimental results show the advantages of applying deep learning methods over state-of-the-art hand-crafted features on the suggested cross-subject and cross-view evaluation criteria for our dataset. The introduction of this large scale dataset will enable the community to apply, develop and adapt various data-hungry learning techniques for the task of depth-based and RGB+D-based human activity analysis.

10 pages

References in corpus (8)

• Sequence to Sequence Learning with Neural Networks
• Visualizing and Understanding Recurrent Networks
• Co-occurrence Feature Learning for Skeleton based Action Recognition using Regularized Deep LSTM Networks
• 3D Human Activity Recognition with Reconfigurable Convolutional Neural Networks
• Differential Recurrent Neural Networks for Action Recognition
• Cross-view Action Modeling, Learning and Recognition
• Space-Time Representation of People Based on 3D Skeletal Data: A Review
• Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos

Cited by in corpus (78)

• Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
• An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data
• SkeleMotion: A New Representation of Skeleton Joint Sequences Based on Motion Information for 3D Action Recognition
• PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding
• Two-Stream 3D Convolutional Neural Network for Skeleton-Based Action Recognition
• An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition
• Unsupervised Learning of View-invariant Action Representations
• A Survey on 3D Skeleton-Based Action Recognition Using Learning Method
• PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences
• Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation
• Action-Attending Graphic Neural Network
• View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data
• Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences
• Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition
• Skeleton-based Action Recognition of People Handling Objects
• Learning to Recognize 3D Human Action from A New Skeleton-based Representation Using Deep Convolutional Neural Networks
• Multi-Modality Fusion based on Consensus-Voting and 3D Convolution for Isolated Gesture Recognition
• RGB-D-based Human Motion Recognition with Deep Learning: A Survey
• Learning in an Uncertain World: Representing Ambiguity Through Multiple Hypotheses
• Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning
• Spatio-Temporal Graph Convolution for Skeleton Based Action Recognition
• Unsupervised Learning of Long-Term Motion Dynamics for Videos
• Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks
• Action Machine: Rethinking Action Recognition in Trimmed Videos
• Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching
• Memory Attention Networks for Skeleton-based Action Recognition
• Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition
• Exploiting the ConvLSTM: Human Action Recognition using Raw Depth Video-Based Recurrent Neural Networks
• Graph Edge Convolutional Neural Networks for Skeleton Based Action Recognition
• First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations
• Learning Human Motion Models for Long-term Predictions
• Video Representation Learning Using Discriminative Pooling
• Skepxels: Spatio-temporal Image Representation of Human Skeleton Joints for Action Recognition
• Depth Pooling Based Large-scale 3D Action Recognition with Convolutional Neural Networks
• Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos
• Privacy-Preserving Action Recognition for Smart Hospitals using Low-Resolution Depth Images
• Deep Learning on Lie Groups for Skeleton-based Action Recognition
• Human Action Generation with Generative Adversarial Networks
• Tensor graph convolutional neural network
• Concurrence-Aware Long Short-Term Sub-Memories for Person-Person Action Recognition
• Learning Multi-Granular Spatio-Temporal Graph Network for Skeleton-based Action Recognition
• Action Recognition for Depth Video using Multi-view Dynamic Images
• Pose-Based Two-Stream Relational Networks for Action Recognition in Videos
• Learning Human Pose Models from Synthesized Data for Robust RGB-D Action Recognition
• Experimental Force-Torque Dataset for Robot Learning of Multi-Shape Insertion
• JOLO-GCN: Mining Joint-Centered Light-Weight Information for Skeleton-Based Action Recognition
• Iterate & Cluster: Iterative Semi-Supervised Action Recognition
• Robust 3D Action Recognition through Sampling Local Appearances and Global Distributions
• Classifying action correctness in physical rehabilitation exercises
• Large-scale Isolated Gesture Recognition Using Convolutional Neural Networks
• RGB-D Based Action Recognition with Light-weight 3D Convolutional Networks
• Fine-grained Event Learning of Human-Object Interaction with LSTM-CRF
• CNN-based Action Recognition and Supervised Domain Adaptation on 3D Body Skeletons via Kernel Feature Maps
• Viewpoint Invariant Action Recognition using RGB-D Videos
• Online Action Recognition based on Incremental Learning of Weighted Covariance Descriptors
• Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition
• Multi-Scale Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition
• Quaternion Product Units for Deep Learning on 3D Rotation Groups
• Graph Distillation for Action Detection with Privileged Modalities
• Modality Distillation with Multiple Stream Networks for Action Recognition
• Time-Aware and View-Aware Video Rendering for Unsupervised Representation Learning
• Action Recognition with Spatio-Temporal Visual Attention on Skeleton Image Sequences
• STS Classification with Dual-stream CNN
• Deep-Temporal LSTM for Daily Living Action Recognition
• Unsupervised Feature Learning of Human Actions as Trajectories in Pose Embedding Manifold
• Adaptive Discriminative Regularization for Visual Classification
• LSTA-Net: Long short-term Spatio-Temporal Aggregation Network for Skeleton-based Action Recognition
• Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection
• Learning event representation: As sparse as possible, but not sparser
• Classification of sparsely labeled spatio-temporal data through semi-supervised adversarial learning
• Empowering cyberphysical systems of systems with intelligence
• RFC-HyPGCN: A Runtime Sparse Feature Compress Accelerator for Skeleton-Based GCNs Action Recognition Model with Hybrid Pruning
• Signal Alignment for Humanoid Skeletons via the Globally Optimal Reparameterization Algorithm
• Scene Flow to Action Map: A New Representation for RGB-D based Action Recognition with Convolutional Neural Networks
• Spatio-Temporal Inception Graph Convolutional Networks for Skeleton-Based Action Recognition
• Learning Chebyshev Basis in Graph Convolutional Networks for Skeleton-based Action Recognition
• Interpretable 3D Human Action Analysis with Temporal Convolutional Networks
• Train, Diagnose and Fix: Interpretable Approach for Fine-grained Action Recognition