Object-Centric Image Generation with Factored Depths, Locations, and Appearances

Abstract

We present a generative model of images that explicitly reasons over the set of objects they show. Our model learns a structured latent representation that separates objects from each other and from the background; unlike prior works, it explicitly represents the 2D position and depth of each object, as well as an embedding of its segmentation mask and appearance. The model can be trained from images alone in a purely unsupervised fashion without the need for object masks or depth information. Moreover, it always generates complete objects, even though a significant fraction of training images contain occlusions. Finally, we show that our model can infer decompositions of novel images into their constituent objects, including accurate prediction of depth ordering and segmentation of occluded parts.

References in corpus (9)

• Auto-Encoding Variational Bayes
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Categorical Reparameterization with Gumbel-Softmax
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
• Multi-Object Representation Learning with Iterative Variational Inference
• MONet: Unsupervised Scene Decomposition and Representation
• GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
• Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction

Cited by in corpus (6)

• BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
• RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces
• GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
• Compositional Transformers for Scene Generation
• Unsupervised object-centric video generation and decomposition in 3D
• Illiterate DALL-E Learns to Compose