GRAINS: Generative Recursive Autoencoders for INdoor Scenes

Abstract

We present a generative neural network which enables us to generate plausible 3D indoor scenes in large quantities and varieties, easily and highly efficiently. Our key observation is that indoor scene structures are inherently hierarchical. Hence, our network is not convolutional; it is a recursive neural network or RvNN. Using a dataset of annotated scene hierarchies, we train a variational recursive autoencoder, or RvNN-VAE, which performs scene object grouping during its encoding phase and scene generation during decoding. Specifically, a set of encoders are recursively applied to group 3D objects based on support, surround, and co-occurrence relations in a scene, encoding information about object spatial properties, semantics, and their relative positioning with respect to other objects in the hierarchy. By training a variational autoencoder (VAE), the resulting fixed-length codes roughly follow a Gaussian distribution. A novel 3D scene can be generated hierarchically by the decoder from a randomly sampled code from the learned distribution. We coin our method GRAINS, for Generative Recursive Autoencoders for INdoor Scenes. We demonstrate the capability of GRAINS to generate plausible and diverse 3D indoor scenes and compare with existing methods for 3D scene synthesis. We show applications of GRAINS including 3D scene modeling from 2D layouts, scene editing, and semantic scene segmentation via PointNet whose performance is boosted by the large quantity and variety of 3D scenes generated by our method.

21 pages, 26 figures

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Convolutional Networks on Graphs for Learning Molecular Fingerprints
• Learning Convolutional Neural Networks for Graphs
• Human-centric Indoor Scene Synthesis Using Stochastic Grammar

Cited by in corpus (9)

• VRCopilot: Authoring 3D Layouts with Generative AI Models in VR
• ClipGen: A Deep Generative Model for Clipart Vectorization and Synthesis
• Scene Synthesis from Human Motion
• LayoutEnhancer: Generating Good Indoor Layouts from Imperfect Data
• Diverse Multimedia Layout Generation with Multi Choice Learning
• Structured Generative Models for Scene Understanding
• RLSS: A Deep Reinforcement Learning Algorithm for Sequential Scene Generation
• Intelligent Generation of Graphical Game Assets: A Conceptual Framework and Systematic Review of the State of the Art
• PartNet: A Recursive Part Decomposition Network for Fine-grained and Hierarchical Shape Segmentation