Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis

Abstract

In recent years, Generative Adversarial Networks have achieved impressive results in photorealistic image synthesis. This progress nurtures hopes that one day the classical rendering pipeline can be replaced by efficient models that are learned directly from images. However, current image synthesis models operate in the 2D domain where disentangling 3D properties such as camera viewpoint or object pose is challenging. Furthermore, they lack an interpretable and controllable representation. Our key hypothesis is that the image generation process should be modeled in 3D space as the physical world surrounding us is intrinsically three-dimensional. We define the new task of 3D controllable image synthesis and propose an approach for solving it by reasoning both in 3D space and in the 2D image domain. We demonstrate that our model is able to disentangle latent 3D factors of simple multi-object scenes in an unsupervised fashion from raw images. Compared to pure 2D baselines, it allows for synthesizing scenes that are consistent wrt. changes in viewpoint or object pose. We further evaluate various 3D representations in terms of their usefulness for this challenging task.

arXiv commentCVPR 2020

References in corpus (16)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• Which Training Methods for GANs do actually Converge?
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Multi-Object Representation Learning with Iterative Variational Inference
• Learning What and Where to Draw
• LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation
• MONet: Unsupervised Scene Decomposition and Representation
• The Numerics of GANs
• Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning
• Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
• Visual Object Networks: Image Generation with Disentangled 3D Representation
• Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling
• R-SQAIR: Relational Sequential Attend, Infer, Repeat
• RGBD-GAN: Unsupervised 3D Representation Learning From Natural Image Datasets via RGBD Image Synthesis

Cited by in corpus (2)

• Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data
• RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces