End-to-End Optimization of Scene Layout

Abstract

We propose an end-to-end variational generative model for scene layout synthesis conditioned on scene graphs. Unlike unconditional scene layout generation, we use scene graphs as an abstract but general representation to guide the synthesis of diverse scene layouts that satisfy relationships included in the scene graph. This gives rise to more flexible control over the synthesis process, allowing various forms of inputs such as scene layouts extracted from sentences or inferred from a single color image. Using our conditional layout synthesizer, we can generate various layouts that share the same structure of the input example. In addition to this conditional generation design, we also integrate a differentiable rendering module that enables layout refinement using only 2D projections of the scene. Given a depth and a semantics map, the differentiable rendering module enables optimizing over the synthesized layout to fit the given input in an analysis-by-synthesis fashion. Experiments suggest that our model achieves higher accuracy and diversity in conditional scene synthesis and allows exemplar-based scene generation from various input forms.

arXiv commentCVPR 2020 (Oral). Project page: http://3dsln.csail.mit.edu/

References in corpus (6)

• Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning
• InteriorNet: Mega-scale Multi-sensor Photo-realistic Indoor Scenes Dataset
• Text to 3D Scene Generation with Rich Lexical Grounding
• Learning to Generate and Reconstruct 3D Meshes with only 2D Supervision
• Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling
• Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image