NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding

Abstract

Research on depth-based human activity analysis achieved outstanding performance and demonstrated the effectiveness of 3D representation for action recognition. The existing depth-based and RGB+D-based action recognition benchmarks have a number of limitations, including the lack of large-scale training samples, realistic number of distinct class categories, diversity in camera views, varied environmental conditions, and variety of human subjects. In this work, we introduce a large-scale dataset for RGB+D human action recognition, which is collected from 106 distinct subjects and contains more than 114 thousand video samples and 8 million frames. This dataset contains 120 different action classes including daily, mutual, and health-related activities. We evaluate the performance of a series of existing 3D activity analysis methods on this dataset, and show the advantage of applying deep learning methods for 3D-based human action recognition. Furthermore, we investigate a novel one-shot 3D activity recognition problem on our dataset, and a simple yet effective Action-Part Semantic Relevance-aware (APSR) framework is proposed for this task, which yields promising results for recognition of the novel action classes. We believe the introduction of this large-scale dataset will enable the community to apply, adapt, and develop various data-hungry learning techniques for depth-based and RGB+D-based human activity understanding. [The dataset is available at: http://rose1.ntu.edu.sg/Datasets/actionRecognition.asp]

IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Representation learning for very short texts using weighted word embedding aggregation
• PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding
• When Kernel Methods meet Feature Learning: Log-Covariance Network for Action Recognition from Skeletal Data

Cited by in corpus (32)

• PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences
• Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition
• Decoupled Spatial-Temporal Attention Network for Skeleton-Based Action Recognition
• Bodily Behaviors in Social Interaction: Novel Annotations and State-of-the-Art Evaluation
• UNIK: A Unified Framework for Real-world Skeleton-based Action Recognition
• Learning from Temporal Spatial Cubism for Cross-Dataset Skeleton-based Action Recognition
• Action Capsules: Human Skeleton Action Recognition
• Vertex Feature Encoding and Hierarchical Temporal Modeling in a Spatial-Temporal Graph Convolutional Network for Action Recognition
• Temporal Graph Modeling for Skeleton-based Action Recognition
• Video-based Contrastive Learning on Decision Trees: from Action Recognition to Autism Diagnosis
• Logsig-RNN: a novel network for robust and efficient skeleton-based action recognition
• Prototypical Contrast and Reverse Prediction: Unsupervised Skeleton Based Action Recognition
• IMUTube: Automatic Extraction of Virtual on-body Accelerometry from Video for Human Activity Recognition
• DMCL: Distillation Multiple Choice Learning for Multimodal Action Recognition
• Unsupervised Motion Representation Learning with Capsule Autoencoders
• ElderSim: A Synthetic Data Generation Platform for Human Action Recognition in Eldercare Applications
• Learning Multi-Granular Spatio-Temporal Graph Network for Skeleton-based Action Recognition
• Skeleton Focused Human Activity Recognition in RGB Video
• RSA: Randomized Simulation as Augmentation for Robust Human Action Recognition
• JOLO-GCN: Mining Joint-Centered Light-Weight Information for Skeleton-Based Action Recognition
• 3DV: 3D Dynamic Voxel for Action Recognition in Depth Video
• Mix Dimension in Poincaré Geometry for 3D Skeleton-based Action Recognition
• Multi-Scale Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition
• Bonn Activity Maps: Dataset Description
• Improving Skeleton-based Action Recognitionwith Robust Spatial and Temporal Features
• AnimGAN: A Spatiotemporally-Conditioned Generative Adversarial Network for Character Animation
• Spatio-Temporal Dual Affine Differential Invariant for Skeleton-based Action Recognition
• LSTA-Net: Long short-term Spatio-Temporal Aggregation Network for Skeleton-based Action Recognition
• Spatio-Temporal Graph Scattering Transform
• Feature-Supervised Action Modality Transfer
• Divide and Rule: Recurrent Partitioned Network for Dynamic Processes
• Hierarchical Action Classification with Network Pruning