Learning to Infer and Execute 3D Shape Programs

Abstract

Human perception of 3D shapes goes beyond reconstructing them as a set of points or a composition of geometric primitives: we also effortlessly understand higher-level shape structure such as the repetition and reflective symmetry of object parts. In contrast, recent advances in 3D shape sensing focus more on low-level geometry but less on these higher-level relationships. In this paper, we propose 3D shape programs, integrating bottom-up recognition systems with top-down, symbolic program structure to capture both low-level geometry and high-level structural priors for 3D shapes. Because there are no annotations of shape programs for real shapes, we develop neural modules that not only learn to infer 3D shape programs from raw, unannotated shapes, but also to execute these programs for shape reconstruction. After initial bootstrapping, our end-to-end differentiable model learns 3D shape programs by reconstructing shapes in a self-supervised manner. Experiments demonstrate that our model accurately infers and executes 3D shape programs for highly complex shapes from various categories. It can also be integrated with an image-to-shape module to infer 3D shape programs directly from an RGB image, leading to 3D shape reconstructions that are both more accurate and more physically plausible.

arXiv commentICLR 2019. Project page: http://shape2prog.csail.mit.edu

References in corpus (11)

• Deep Convolutional Inverse Graphics Network
• MarrNet: 3D Shape Reconstruction via 2.5D Sketches
• DeepCoder: Learning to Write Programs
• Neural Programmer-Interpreters
• Synthesizing Programs for Images using Reinforced Adversarial Learning
• RobustFill: Neural Program Learning under Noisy I/O
• Neuro-Symbolic Program Synthesis
• Making Neural Programming Architectures Generalize via Recursion
• Programming with a Differentiable Forth Interpreter
• TerpreT: A Probabilistic Programming Language for Program Induction
• Neurally-Guided Procedural Models: Amortized Inference for Procedural Graphics Programs using Neural Networks

Cited by in corpus (26)

• Joint Hand-object 3D Reconstruction from a Single Image with Cross-branch Feature Fusion
• StructureNet: Hierarchical Graph Networks for 3D Shape Generation
• babble: Learning Better Abstractions with E-Graphs and Anti-Unification
• Reconstructing editable prismatic CAD from rounded voxel models
• A Survey on Verification and Validation, Testing and Evaluations of Neurosymbolic Artificial Intelligence
• ShapeMOD: Macro Operation Discovery for 3D Shape Programs
• Write, Execute, Assess: Program Synthesis with a REPL
• ProTo: Program-Guided Transformer for Program-Guided Tasks
• ImageEye: Batch Image Processing Using Program Synthesis
• Learning to simulate and design for structural engineering
• Fusion 360 Gallery: A Dataset and Environment for Programmatic CAD Construction from Human Design Sequences
• A Survey of Methods for Converting Unstructured Data to CSG Models
• Modeling 3D Shapes by Reinforcement Learning
• Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image
• PT2PC: Learning to Generate 3D Point Cloud Shapes from Part Tree Conditions
• Semantic Correspondence via 2D-3D-2D Cycle
• Learning Manifold Patch-Based Representations of Man-Made Shapes
• A Neural-Symbolic Architecture for Inverse Graphics Improved by Lifelong Meta-Learning
• Program Synthesis Guided Reinforcement Learning for Partially Observed Environments
• Adaptive Procedural Task Generation for Hard-Exploration Problems
• LAVAE: Disentangling Location and Appearance
• Understanding Pixel-level 2D Image Semantics with 3D Keypoint Knowledge Engine
• On the Complexity of the CSG Tree Extraction Problem
• A Neural-Symbolic Framework for Mental Simulation
• Unsupervised Program Synthesis for Images By Sampling Without Replacement
• Latent Execution for Neural Program Synthesis