3D-Aware Scene Manipulation via Inverse Graphics

Abstract

We aim to obtain an interpretable, expressive, and disentangled scene representation that contains comprehensive structural and textural information for each object. Previous scene representations learned by neural networks are often uninterpretable, limited to a single object, or lacking 3D knowledge. In this work, we propose 3D scene de-rendering networks (3D-SDN) to address the above issues by integrating disentangled representations for semantics, geometry, and appearance into a deep generative model. Our scene encoder performs inverse graphics, translating a scene into a structured object-wise representation. Our decoder has two components: a differentiable shape renderer and a neural texture generator. The disentanglement of semantics, geometry, and appearance supports 3D-aware scene manipulation, e.g., rotating and moving objects freely while keeping the consistent shape and texture, and changing the object appearance without affecting its shape. Experiments demonstrate that our editing scheme based on 3D-SDN is superior to its 2D counterpart.

arXiv commentNeurIPS 2018. Code: https://github.com/ysymyth/3D-SDN Website: http://3dsdn.csail.mit.edu/

References in corpus (3)

• Conditional Generative Adversarial Nets
• Toward Multimodal Image-to-Image Translation
• PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications

Cited by in corpus (19)

• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Differentiable Surface Splatting for Point-based Geometry Processing
• Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
• BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
• Learning Physical Graph Representations from Visual Scenes
• Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis
• Towards Realistic 3D Embedding via View Alignment
• UGAN: Untraceable GAN for Multi-Domain Face Translation
• AdvSPADE: Realistic Unrestricted Attacks for Semantic Segmentation
• MichiGAN: Multi-Input-Conditioned Hair Image Generation for Portrait Editing
• A Neural-Symbolic Architecture for Inverse Graphics Improved by Lifelong Meta-Learning
• PerMO: Perceiving More at Once from a Single Image for Autonomous Driving
• Generative Hierarchical Models for Parts, Objects, and Scenes
• Learning to Manipulate Individual Objects in an Image
• Semantic Image Manipulation Using Scene Graphs
• Deep Feedback Inverse Problem Solver
• Learning Scene Dynamics from Point Cloud Sequences
• Robust 2D/3D Vehicle Parsing in CVIS
• Learning Equivariant Representations