Rotation Equivariant 3D Hand Mesh Generation from a Single RGB Image

Abstract

We develop a rotation equivariant model for generating 3D hand meshes from 2D RGB images. This guarantees that as the input image of a hand is rotated the generated mesh undergoes a corresponding rotation. Furthermore, this removes undesirable deformations in the meshes often generated by methods without rotation equivariance. By building a rotation equivariant model, through considering symmetries in the problem, we reduce the need for training on very large datasets to achieve good mesh reconstruction. The encoder takes images defined on ${\mathbb{Z}}^2$ and maps these to latent functions defined on the group $C_8$. We introduce a novel vector mapping function to map the function defined on $C_8$ to a latent point cloud space defined on the group $\mathrm{SO}(2)$. Further, we introduce a 3D projection function that learns a 3D function from the $\mathrm{SO}(2)$ latent space. Finally, we use an $\mathrm{SO}(3)$ equivariant decoder to ensure rotation equivariance. Our rotation equivariant model outperforms state-of-the-art methods on a real-world dataset and we demonstrate that it accurately captures the shape and pose in the generated meshes under rotation of the input hand.

References in corpus (23)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Semi-Supervised Classification with Graph Convolutional Networks
• Neural Message Passing for Quantum Chemistry
• Geometric deep learning: going beyond Euclidean data
• Embodied Hands: Modeling and Capturing Hands and Bodies Together
• MeshCNN: A Network with an Edge
• Group Equivariant Convolutional Networks
• Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds
• Simple and Deep Graph Convolutional Networks
• SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks
• On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups
• 3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data
• Exploiting Cyclic Symmetry in Convolutional Neural Networks
• Equivariance Through Parameter-Sharing
• E(n) Equivariant Graph Neural Networks
• General $E(2)$-Equivariant Steerable CNNs
• Equivariant Flows: sampling configurations for multi-body systems with symmetric energies
• Intertwiners between Induced Representations (with Applications to the Theory of Equivariant Neural Networks)
• A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups
• A Wigner-Eckart Theorem for Group Equivariant Convolution Kernels
• Single Image 3D Hand Reconstruction with Mesh Convolutions
• A Graph VAE and Graph Transformer Approach to Generating Molecular Graphs
• Rotation Equivariant Deforestation Segmentation and Driver Classification