StructureNet: Hierarchical Graph Networks for 3D Shape Generation

Abstract

The ability to generate novel, diverse, and realistic 3D shapes along with associated part semantics and structure is central to many applications requiring high-quality 3D assets or large volumes of realistic training data. A key challenge towards this goal is how to accommodate diverse shape variations, including both continuous deformations of parts as well as structural or discrete alterations which add to, remove from, or modify the shape constituents and compositional structure. Such object structure can typically be organized into a hierarchy of constituent object parts and relationships, represented as a hierarchy of n-ary graphs. We introduce StructureNet, a hierarchical graph network which (i) can directly encode shapes represented as such n-ary graphs; (ii) can be robustly trained on large and complex shape families; and (iii) can be used to generate a great diversity of realistic structured shape geometries. Technically, we accomplish this by drawing inspiration from recent advances in graph neural networks to propose an order-invariant encoding of n-ary graphs, considering jointly both part geometry and inter-part relations during network training. We extensively evaluate the quality of the learned latent spaces for various shape families and show significant advantages over baseline and competing methods. The learned latent spaces enable several structure-aware geometry processing applications, including shape generation and interpolation, shape editing, or shape structure discovery directly from un-annotated images, point clouds, or partial scans.

Conditionally Accepted to Siggraph Asia 2019

References in corpus (12)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Auto-Encoding Variational Bayes
• Inductive Representation Learning on Large Graphs
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Learning Representations and Generative Models for 3D Point Clouds
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models
• Learning Deep Generative Models of Graphs
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• Structured Generative Models of Natural Source Code
• Learning to Infer and Execute 3D Shape Programs
• PartNet: A Recursive Part Decomposition Network for Fine-grained and Hierarchical Shape Segmentation

Cited by in corpus (15)

• Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple Images
• ShapeAssembly: Learning to Generate Programs for 3D Shape Structure Synthesis
• Approximate Convex Decomposition for 3D Meshes with Collision-Aware Concavity and Tree Search
• Unsupervised Learning for Cuboid Shape Abstraction via Joint Segmentation from Point Clouds
• Towards Reusable Surrogate Models: Graph-Based Transfer Learning on Trusses
• NeuralTailor: Reconstructing Sewing Pattern Structures from 3D Point Clouds of Garments
• MatFormer: A Generative Model for Procedural Materials
• ShapeMOD: Macro Operation Discovery for 3D Shape Programs
• OpenECAD: An Efficient Visual Language Model for Editable 3D-CAD Design
• Learning Mesh Representations via Binary Space Partitioning Tree Networks
• TilinGNN: Learning to Tile with Self-Supervised Graph Neural Network
• Scan2Part: Fine-grained and Hierarchical Part-level Understanding of Real-World 3D Scans
• A Revisit of Shape Editing Techniques: from the Geometric to the Neural Viewpoint
• Single Image 3D Object Estimation with Primitive Graph Networks
• Towards Part-Based Understanding of RGB-D Scans