DeepHPS: End-to-end Estimation of 3D Hand Pose and Shape by Learning from Synthetic Depth

Abstract

Articulated hand pose and shape estimation is an important problem for vision-based applications such as augmented reality and animation. In contrast to the existing methods which optimize only for joint positions, we propose a fully supervised deep network which learns to jointly estimate a full 3D hand mesh representation and pose from a single depth image. To this end, a CNN architecture is employed to estimate parametric representations i.e. hand pose, bone scales and complex shape parameters. Then, a novel hand pose and shape layer, embedded inside our deep framework, produces 3D joint positions and hand mesh. Lack of sufficient training data with varying hand shapes limits the generalized performance of learning based methods. Also, manually annotating real data is suboptimal. Therefore, we present SynHand5M: a million-scale synthetic dataset with accurate joint annotations, segmentation masks and mesh files of depth maps. Among model based learning (hybrid) methods, we show improved results on our dataset and two of the public benchmarks i.e. NYU and ICVL. Also, by employing a joint training strategy with real and synthetic data, we recover 3D hand mesh and pose from real images in 3.7ms.

arXiv commentAccepted for publication in 3DV-2018 (http://3dv18.uniud.it/)

References in corpus (18)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• Embodied Hands: Modeling and Capturing Hands and Bodies Together
• Hands Deep in Deep Learning for Hand Pose Estimation
• Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
• Pose Guided Structured Region Ensemble Network for Cascaded Hand Pose Estimation
• Hand3D: Hand Pose Estimation using 3D Neural Network
• Robust 3D Hand Pose Estimation in Single Depth Images: from Single-View CNN to Multi-View CNNs
• V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map
• BigHand2.2M Benchmark: Hand Pose Dataset and State of the Art Analysis
• Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Input
• SurfNet: Generating 3D shape surfaces using deep residual networks
• Training a Feedback Loop for Hand Pose Estimation
• Learning to Search on Manifolds for 3D Pose Estimation of Articulated Objects
• Occlusion-aware Hand Pose Estimation Using Hierarchical Mixture Density Network
• Simultaneous Hand Pose and Skeleton Bone-Lengths Estimation from a Single Depth Image
• GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB
• Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals
• Dense 3D Regression for Hand Pose Estimation