V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map

Abstract

Most of the existing deep learning-based methods for 3D hand and human pose estimation from a single depth map are based on a common framework that takes a 2D depth map and directly regresses the 3D coordinates of keypoints, such as hand or human body joints, via 2D convolutional neural networks (CNNs). The first weakness of this approach is the presence of perspective distortion in the 2D depth map. While the depth map is intrinsically 3D data, many previous methods treat depth maps as 2D images that can distort the shape of the actual object through projection from 3D to 2D space. This compels the network to perform perspective distortion-invariant estimation. The second weakness of the conventional approach is that directly regressing 3D coordinates from a 2D image is a highly non-linear mapping, which causes difficulty in the learning procedure. To overcome these weaknesses, we firstly cast the 3D hand and human pose estimation problem from a single depth map into a voxel-to-voxel prediction that uses a 3D voxelized grid and estimates the per-voxel likelihood for each keypoint. We design our model as a 3D CNN that provides accurate estimates while running in real-time. Our system outperforms previous methods in almost all publicly available 3D hand and human pose estimation datasets and placed first in the HANDS 2017 frame-based 3D hand pose estimation challenge. The code is available in https://github.com/mks0601/V2V-PoseNet_RELEASE.

HANDS 2017 Challenge Frame-based 3D Hand Pose Estimation Winner (ICCV 2017), Published at CVPR 2018

References in corpus (9)

• Stacked Hourglass Networks for Human Pose Estimation
• Pose Guided Structured Region Ensemble Network for Cascaded Hand Pose Estimation
• Hand3D: Hand Pose Estimation using 3D Neural Network
• Robust 3D Hand Pose Estimation in Single Depth Images: from Single-View CNN to Multi-View CNNs
• The 2017 Hands in the Million Challenge on 3D Hand Pose Estimation
• Towards Good Practices for Deep 3D Hand Pose Estimation
• End-to-end Global to Local CNN Learning for Hand Pose Recovery in Depth Data
• BigHand2.2M Benchmark: Hand Pose Dataset and State of the Art Analysis
• First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations

Cited by in corpus (7)

• Model-based Hand Pose Estimation for Generalized Hand Shape with Appearance Normalization
• Measuring Generalisation to Unseen Viewpoints, Articulations, Shapes and Objects for 3D Hand Pose Estimation under Hand-Object Interaction
• DeepHPS: End-to-end Estimation of 3D Hand Pose and Shape by Learning from Synthetic Depth
• JGR-P2O: Joint Graph Reasoning based Pixel-to-Offset Prediction Network for 3D Hand Pose Estimation from a Single Depth Image
• Point-to-Pose Voting based Hand Pose Estimation using Residual Permutation Equivariant Layer
• An End-to-end Framework for Unconstrained Monocular 3D Hand Pose Estimation
• MURAUER: Mapping Unlabeled Real Data for Label AUstERity