Single Image 3D Hand Reconstruction with Mesh Convolutions

Abstract

Monocular 3D reconstruction of deformable objects, such as human body parts, has been typically approached by predicting parameters of heavyweight linear models. In this paper, we demonstrate an alternative solution that is based on the idea of encoding images into a latent non-linear representation of meshes. The prior on 3D hand shapes is learned by training an autoencoder with intrinsic graph convolutions performed in the spectral domain. The pre-trained decoder acts as a non-linear statistical deformable model. The latent parameters that reconstruct the shape and articulated pose of hands in the image are predicted using an image encoder. We show that our system reconstructs plausible meshes and operates in real-time. We evaluate the quality of the mesh reconstructions produced by the decoder on a new dataset and show latent space interpolation results. Our code, data, and models will be made publicly available.

arXiv commentProceedings of the British Machine Vision Conference (BMVC 2019)

References in corpus (18)

• Geometric deep learning: going beyond Euclidean data
• Densely Connected Convolutional Networks
• Embodied Hands: Modeling and Capturing Hands and Bodies Together
• Hierarchical Graph Representation Learning with Differentiable Pooling
• Towards Sparse Hierarchical Graph Classifiers
• Learning shape correspondence with anisotropic convolutional neural networks
• End-to-end Recovery of Human Shape and Pose
• MeshGAN: Non-linear 3D Morphable Models of Faces
• Unite the People: Closing the Loop Between 3D and 2D Human Representations
• Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation
• Geometric deep learning on graphs and manifolds using mixture model CNNs
• Generating 3D faces using Convolutional Mesh Autoencoders
• Monocular Total Capture: Posing Face, Body, and Hands in the Wild
• Learning to Estimate 3D Human Pose and Shape from a Single Color Image
• End-to-end Hand Mesh Recovery from a Monocular RGB Image
• Dense 3D Face Decoding over 2500FPS: Joint Texture & Shape Convolutional Mesh Decoders
• Photo Wake-Up: 3D Character Animation from a Single Photo
• A Simple Approach to Intrinsic Correspondence Learning on Unstructured 3D Meshes

Cited by in corpus (6)

• HandTailor: Towards High-Precision Monocular 3D Hand Recovery
• One-stage Shape Instantiation from a Single 2D Image to 3D Point Cloud
• An End-to-end Framework for Unconstrained Monocular 3D Hand Pose Estimation
• Monocular Human Shape and Pose with Dense Mesh-borne Local Image Features
• Instantiation-Net: 3D Mesh Reconstruction from Single 2D Image for Right Ventricle
• Rotation Equivariant 3D Hand Mesh Generation from a Single RGB Image