First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations

Abstract

In this work we study the use of 3D hand poses to recognize first-person dynamic hand actions interacting with 3D objects. Towards this goal, we collected RGB-D video sequences comprised of more than 100K frames of 45 daily hand action categories, involving 26 different objects in several hand configurations. To obtain hand pose annotations, we used our own mo-cap system that automatically infers the 3D location of each of the 21 joints of a hand model via 6 magnetic sensors and inverse kinematics. Additionally, we recorded the 6D object poses and provide 3D object models for a subset of hand-object interaction sequences. To the best of our knowledge, this is the first benchmark that enables the study of first-person hand actions with the use of 3D hand poses. We present an extensive experimental evaluation of RGB-D and pose-based action recognition by 18 baselines/state-of-the-art approaches. The impact of using appearance features, poses, and their combinations are measured, and the different training/testing protocols are evaluated. Finally, we assess how ready the 3D hand pose estimation field is when hands are severely occluded by objects in egocentric views and its influence on action recognition. From the results, we see clear benefits of using hand pose as a cue for action recognition compared to other data modalities. Our dataset and experiments can be of interest to communities of 3D hand pose estimation, 6D object pose, and robotics as well as action recognition.

arXiv commentAccepted to CVPR 2018

References in corpus (10)

• NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis
• Convolutional Two-Stream Network Fusion for Video Action Recognition
• Capturing Hands in Action using Discriminative Salient Points and Physics Simulation
• Convolutional Pose Machines
• Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
• Differential Recurrent Neural Networks for Action Recognition
• 3D Hand Pose Detection in Egocentric RGB-D Images
• The 2017 Hands in the Million Challenge on 3D Hand Pose Estimation
• BigHand2.2M Benchmark: Hand Pose Dataset and State of the Art Analysis
• Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Input

Cited by in corpus (6)

• V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map
• Occlusion-aware Hand Pose Estimation Using Hierarchical Mixture Density Network
• Model-based Hand Pose Estimation for Generalized Hand Shape with Appearance Normalization
• Understanding Human Hands in Contact at Internet Scale
• Quaternion Product Units for Deep Learning on 3D Rotation Groups
• Egocentric 6-DoF Tracking of Small Handheld Objects