Rotation Equivariant 3D Hand Mesh Generation from a Single RGB Image

Abstract

We develop a rotation equivariant model for generating 3D hand meshes from 2D RGB images. This guarantees that as the input image of a hand is rotated the generated mesh undergoes a corresponding rotation. Furthermore, this removes undesirable deformations in the meshes often generated by methods without rotation equivariance. By building a rotation equivariant model, through considering symmetries in the problem, we reduce the need for training on very large datasets to achieve good mesh reconstruction. The encoder takes images defined on ${\mathbb{Z}}^2$ and maps these to latent functions defined on the group $C_8$. We introduce a novel vector mapping function to map the function defined on $C_8$ to a latent point cloud space defined on the group $\mathrm{SO}(2)$. Further, we introduce a 3D projection function that learns a 3D function from the $\mathrm{SO}(2)$ latent space. Finally, we use an $\mathrm{SO}(3)$ equivariant decoder to ensure rotation equivariance. Our rotation equivariant model outperforms state-of-the-art methods on a real-world dataset and we demonstrate that it accurately captures the shape and pose in the generated meshes under rotation of the input hand.

References in corpus (10)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Semi-Supervised Classification with Graph Convolutional Networks
• Embodied Hands: Modeling and Capturing Hands and Bodies Together
• Simple and Deep Graph Convolutional Networks
• E(n) Equivariant Graph Neural Networks
• Equivariant Flows: sampling configurations for multi-body systems with symmetric energies
• A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups
• A Wigner-Eckart Theorem for Group Equivariant Convolution Kernels
• A Graph VAE and Graph Transformer Approach to Generating Molecular Graphs
• Rotation Equivariant Deforestation Segmentation and Driver Classification