NTU-X: An Enhanced Large-scale Dataset for Improving Pose-based Recognition of Subtle Human Actions

Abstract

The lack of fine-grained joints (facial joints, hand fingers) is a fundamental performance bottleneck for state of the art skeleton action recognition models. Despite this bottleneck, community's efforts seem to be invested only in coming up with novel architectures. To specifically address this bottleneck, we introduce two new pose based human action datasets - NTU60-X and NTU120-X. Our datasets extend the largest existing action recognition dataset, NTU-RGBD. In addition to the 25 body joints for each skeleton as in NTU-RGBD, NTU60-X and NTU120-X dataset includes finger and facial joints, enabling a richer skeleton representation. We appropriately modify the state of the art approaches to enable training using the introduced datasets. Our results demonstrate the effectiveness of these NTU-X datasets in overcoming the aforementioned bottleneck and improve state of the art performance, overall and on previously worst performing action categories. Code and pretrained models can be found at https://github.com/skelemoa/ntu-x .

First two authors contributed equally. Code repository at https://github.com/skelemoa/ntu-x

References in corpus (6)

• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
• Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
• Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition
• A Survey on 3D Skeleton-Based Action Recognition Using Learning Method
• Cross-view Action Modeling, Learning and Recognition
• A Large-scale Varying-view RGB-D Action Dataset for Arbitrary-view Human Action Recognition