Model-based 3D Hand Reconstruction via Self-Supervised Learning

Abstract

Reconstructing a 3D hand from a single-view RGB image is challenging due to various hand configurations and depth ambiguity. To reliably reconstruct a 3D hand from a monocular image, most state-of-the-art methods heavily rely on 3D annotations at the training stage, but obtaining 3D annotations is expensive. To alleviate reliance on labeled training data, we propose S2HAND, a self-supervised 3D hand reconstruction network that can jointly estimate pose, shape, texture, and the camera viewpoint. Specifically, we obtain geometric cues from the input image through easily accessible 2D detected keypoints. To learn an accurate hand reconstruction model from these noisy geometric cues, we utilize the consistency between 2D and 3D representations and propose a set of novel losses to rationalize outputs of the neural network. For the first time, we demonstrate the feasibility of training an accurate 3D hand reconstruction network without relying on manual annotations. Our experiments show that the proposed method achieves comparable performance with recent fully-supervised methods while using fewer supervision data.

Accepted by CVPR21

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Embodied Hands: Modeling and Capturing Hands and Bodies Together
• Stacked Hourglass Networks for Human Pose Estimation
• Joint Hand-object 3D Reconstruction from a Single Image with Cross-branch Feature Fusion
• Robust 3D Hand Pose Estimation in Single Depth Images: from Single-View CNN to Multi-View CNNs
• End-to-end Hand Mesh Recovery from a Monocular RGB Image
• Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz
• Measuring Generalisation to Unseen Viewpoints, Articulations, Shapes and Objects for 3D Hand Pose Estimation under Hand-Object Interaction
• Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstruction