GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations

Abstract

Generative latent-variable models are emerging as promising tools in robotics and reinforcement learning. Yet, even though tasks in these domains typically involve distinct objects, most state-of-the-art generative models do not explicitly capture the compositional nature of visual scenes. Two recent exceptions, MONet and IODINE, decompose scenes into objects in an unsupervised fashion. Their underlying generative processes, however, do not account for component interactions. Hence, neither of them allows for principled sampling of novel scenes. Here we present GENESIS, the first object-centric generative model of 3D visual scenes capable of both decomposing and generating scenes by capturing relationships between scene components. GENESIS parameterises a spatial GMM over images which is decoded from a set of object-centric latent variables that are either inferred sequentially in an amortised fashion or sampled from an autoregressive prior. We train GENESIS on several publicly available datasets and evaluate its performance on scene generation, decomposition, and semi-supervised learning.

arXiv commentPublished at the International Conference on Learning Representations (ICLR) 2020

References in corpus (12)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Auto-Encoding Variational Bayes
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Multi-Object Representation Learning with Iterative Variational Inference
• MONet: Unsupervised Scene Decomposition and Representation
• Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
• Are Disentangled Representations Helpful for Abstract Visual Reasoning?
• Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
• Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs
• Object Discovery with a Copy-Pasting GAN
• Efficient inference in occlusion-aware generative models of images
• Multi-objects Generation with Amortized Structural Regularization

Cited by in corpus (47)

• Object-Centric Learning with Slot Attention
• BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
• Localizing Objects with Self-Supervised Transformers and no Labels
• SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition
• On the Binding Problem in Artificial Neural Networks
• Learning Physical Graph Representations from Visual Scenes
• RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces
• Stacked Capsule Autoencoders
• From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence
• Discrete and continuous representations and processing in deep learning: Looking forward
• Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation
• NeRF-VAE: A Geometry Aware 3D Scene Generative Model
• Unsupervised Discovery of Object Radiance Fields
• Structured Object-Aware Physics Prediction for Video Modeling and Planning
• Generative Adversarial Transformers
• GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
• Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs
• Unsupervised Learning of Compositional Energy Concepts
• Compositional Transformers for Scene Generation
• SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition
• Generative Neurosymbolic Machines
• Forward Prediction for Physical Reasoning
• Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction
• Unsupervised Video Decomposition using Spatio-temporal Iterative Inference
• Object-Centric Image Generation with Factored Depths, Locations, and Appearances
• Reconstruction Bottlenecks in Object-Centric Generative Models
• Self-supervised Video Object Segmentation by Motion Grouping
• Generalization and Robustness Implications in Object-Centric Learning
• Unsupervised object-centric video generation and decomposition in 3D
• Benchmarking Unsupervised Object Representations for Video Sequences
• SLASH: Embracing Probabilistic Circuits into Neural Answer Set Programming
• Unsupervised Part Discovery from Contrastive Reconstruction
• Grounding Spatio-Temporal Language with Transformers
• Causal World Models by Unsupervised Deconfounding of Physical Dynamics
• Towards a Neural Graphics Pipeline for Controllable Image Generation
• PatchNet: Unsupervised Object Discovery based on Patch Embedding
• Knowledge-Guided Object Discovery with Acquired Deep Impressions
• Jointly Learning Truth-Conditional Denotations and Groundings using Parallel Attention
• Language-Mediated, Object-Centric Representation Learning
• Unsupervised Object Learning via Common Fate
• Polyline Generative Navigable Space Segmentation for Autonomous Visual Navigation
• Leveraging Hidden Structure in Self-Supervised Learning
• GP-ConvCNP: Better Generalization for Convolutional Conditional Neural Processes on Time Series Data
• Neuro-Symbolic Forward Reasoning
• Illiterate DALL-E Learns to Compose
• Variational State-Space Models for Localisation and Dense 3D Mapping in 6 DoF
• Conditional Object-Centric Learning from Video