End-to-end Hand Mesh Recovery from a Monocular RGB Image

Abstract

In this paper, we present a HAnd Mesh Recovery (HAMR) framework to tackle the problem of reconstructing the full 3D mesh of a human hand from a single RGB image. In contrast to existing research on 2D or 3D hand pose estimation from RGB or/and depth image data, HAMR can provide a more expressive and useful mesh representation for monocular hand image understanding. In particular, the mesh representation is achieved by parameterizing a generic 3D hand model with shape and relative 3D joint angles. By utilizing this mesh representation, we can easily compute the 3D joint locations via linear interpolations between the vertexes of the mesh, while obtain the 2D joint locations with a projection of the 3D joints.To this end, a differentiable re-projection loss can be defined in terms of the derived representations and the ground-truth labels, thus making our framework end-to-end trainable.Qualitative experiments show that our framework is capable of recovering appealing 3D hand mesh even in the presence of severe occlusions.Quantitatively, our approach also outperforms the state-of-the-art methods for both 2D and 3D hand pose estimation from a monocular RGB image on several benchmark datasets.

arXiv comment11 pages;

References in corpus (6)

• Embodied Hands: Modeling and Capturing Hands and Bodies Together
• Associative Embedding: End-to-End Learning for Joint Detection and Grouping
• Face Alignment Across Large Poses: A 3D Solution
• Face Alignment in Full Pose Range: A 3D Total Solution
• Fast and Robust Hand Tracking Using Detection-Guided Optimization
• 3D Hand Pose Tracking and Estimation Using Stereo Matching

Cited by in corpus (8)

• BiHand: Recovering Hand Mesh with Multi-stage Bisected Hourglass Networks
• Single Image 3D Hand Reconstruction with Mesh Convolutions
• Model-based 3D Hand Reconstruction via Self-Supervised Learning
• SeqHAND:RGB-Sequence-Based 3D Hand Pose and Shape Estimation
• Temporal-Aware Self-Supervised Learning for 3D Hand Pose and Mesh Estimation in Videos
• Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements
• MVHM: A Large-Scale Multi-View Hand Mesh Benchmark for Accurate 3D Hand Pose Estimation
• Dual Grid Net: hand mesh vertex regression from single depth maps