PolyGen: An Autoregressive Generative Model of 3D Meshes

Abstract

Polygon meshes are an efficient representation of 3D geometry, and are of central importance in computer graphics, robotics and games development. Existing learning-based approaches have avoided the challenges of working with 3D meshes, instead using alternative object representations that are more compatible with neural architectures and training approaches. We present an approach which models the mesh directly, predicting mesh vertices and faces sequentially using a Transformer-based architecture. Our model can condition on a range of inputs, including object classes, voxels, and images, and because the model is probabilistic it can produce samples that capture uncertainty in ambiguous scenarios. We show that the model is capable of producing high-quality, usable meshes, and establish log-likelihood benchmarks for the mesh-modelling task. We also evaluate the conditional models on surface reconstruction metrics against alternative methods, and demonstrate competitive performance despite not training directly on this task.

References in corpus (11)

• Sequence to Sequence Learning with Neural Networks
• WaveNet: A Generative Model for Raw Audio
• Conditional Image Generation with PixelCNN Decoders
• Pixel Recurrent Neural Networks
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• The Curious Case of Neural Text Degeneration
• Generating Long Sequences with Sparse Transformers
• MADE: Masked Autoencoder for Distribution Estimation
• Efficient Graph Generation with Graph Recurrent Attention Networks
• PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows
• Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling

Cited by in corpus (14)

• Learning Mesh-Based Simulation with Graph Networks
• ShaRF: Shape-conditioned Radiance Fields from a Single View
• MatFormer: A Generative Model for Procedural Materials
• UCSG-Net -- Unsupervised Discovering of Constructive Solid Geometry Tree
• Computer-Aided Design as Language
• Reconstructing editable prismatic CAD from rounded voxel models
• DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation
• ShapeFlow: Learnable Deformations Among 3D Shapes
• Fusion 360 Gallery: A Dataset and Environment for Programmatic CAD Construction from Human Design Sequences
• Combinatorial 3D Shape Generation via Sequential Assembly
• Parallel Synthesis for Autoregressive Speech Generation
• Learning Manifold Patch-Based Representations of Man-Made Shapes
• A Survey on Deep Geometry Learning: From a Representation Perspective
• Vitruvion: A Generative Model of Parametric CAD Sketches