PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding

Abstract

Despite the fact that many 3D human activity benchmarks being proposed, most existing action datasets focus on the action recognition tasks for the segmented videos. There is a lack of standard large-scale benchmarks, especially for current popular data-hungry deep learning based methods. In this paper, we introduce a new large scale benchmark (PKU-MMD) for continuous multi-modality 3D human action understanding and cover a wide range of complex human activities with well annotated information. PKU-MMD contains 1076 long video sequences in 51 action categories, performed by 66 subjects in three camera views. It contains almost 20,000 action instances and 5.4 million frames in total. Our dataset also provides multi-modality data sources, including RGB, depth, Infrared Radiation and Skeleton. With different modalities, we conduct extensive experiments on our dataset in terms of two scenarios and evaluate different methods by various metrics, including a new proposed evaluation protocol 2D-AP. We believe this large-scale dataset will benefit future researches on action detection for the community.

10 pages

References in corpus (6)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Unstructured Human Activity Detection from RGBD Images
• NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis
• Co-occurrence Feature Learning for Skeleton based Action Recognition using Regularized Deep LSTM Networks
• Cross-view Action Modeling, Learning and Recognition
• RGB-D-based Action Recognition Datasets: A Survey

Cited by in corpus (30)

• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
• Human Action Recognition from Various Data Modalities: A Review
• Human Activity Recognition Using Tools of Convolutional Neural Networks: A State of the Art Review, Data Sets, Challenges and Future Prospects
• Infrared and 3D skeleton feature fusion for RGB-D action recognition
• From CNNs to Transformers in Multimodal Human Action Recognition: A Survey
• Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation
• Markerless Multi-view 3D Human Pose Estimation: a survey
• RGB-D-based Human Motion Recognition with Deep Learning: A Survey
• Deep Learning for Vision-based Prediction: A Survey
• UAV-Human: A Large Benchmark for Human Behavior Understanding with Unmanned Aerial Vehicles
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Making the Invisible Visible: Action Recognition Through Walls and Occlusions
• Towards Robust RGB-D Human Mesh Recovery
• EPAM-Net: An Efficient Pose-driven Attention-guided Multimodal Network for Video Action Recognition
• Identifying Visible Actions in Lifestyle Vlogs
• ElderSim: A Synthetic Data Generation Platform for Human Action Recognition in Eldercare Applications
• EV-Action: Electromyography-Vision Multi-Modal Action Dataset
• Delving into 3D Action Anticipation from Streaming Videos
• Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction
• Bonn Activity Maps: Dataset Description
• Graph Distillation for Action Detection with Privileged Modalities
• OneAdapt: Fast Configuration Adaptation for Video Analytics Applications via Backpropagation
• Skeleton-Based Online Action Prediction Using Scale Selection Network
• Attention-Oriented Action Recognition for Real-Time Human-Robot Interaction
• VideoDG: Generalizing Temporal Relations in Videos to Novel Domains
• Modality Compensation Network: Cross-Modal Adaptation for Action Recognition
• Graph recovery from graph wave equation
• Object Properties Inferring from and Transfer for Human Interaction Motions
• Feature-Supervised Action Modality Transfer
• Bridging the gap between Human Action Recognition and Online Action Detection