Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis

Abstract

We introduce DMTet, a deep 3D conditional generative model that can synthesize high-resolution 3D shapes using simple user guides such as coarse voxels. It marries the merits of implicit and explicit 3D representations by leveraging a novel hybrid 3D representation. Compared to the current implicit approaches, which are trained to regress the signed distance values, DMTet directly optimizes for the reconstructed surface, which enables us to synthesize finer geometric details with fewer artifacts. Unlike deep 3D generative models that directly generate explicit representations such as meshes, our model can synthesize shapes with arbitrary topology. The core of DMTet includes a deformable tetrahedral grid that encodes a discretized signed distance function and a differentiable marching tetrahedra layer that converts the implicit signed distance representation to the explicit surface mesh representation. This combination allows joint optimization of the surface geometry and topology as well as generation of the hierarchy of subdivisions using reconstruction and adversarial losses defined explicitly on the surface mesh. Our approach significantly outperforms existing work on conditional shape synthesis from coarse voxel inputs, trained on a dataset of complex 3D animal shapes. Project page: https://nv-tlabs.github.io/DMTet/.

References in corpus (8)

• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Generative and Discriminative Voxel Modeling with Convolutional Neural Networks
• Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
• Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research
• Learning Deformable Tetrahedral Meshes for 3D Reconstruction
• On the Effectiveness of Weight-Encoded Neural Implicit 3D Shapes
• DeepSpline: Data-Driven Reconstruction of Parametric Curves and Surfaces
• Fast Interactive Object Annotation with Curve-GCN

Cited by in corpus (5)

• Flexible Isosurface Extraction for Gradient-Based Mesh Optimization
• Differential Walk on Spheres
• EASI-Tex: Edge-Aware Mesh Texturing from Single Image
• DiffSound: Differentiable Modal Sound Rendering and Inverse Rendering for Diverse Inference Tasks
• ESCT3D: Efficient and Selectively Controllable Text-Driven 3D Content Generation with Gaussian Splatting