RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces

Abstract

We present RELATE, a model that learns to generate physically plausible scenes and videos of multiple interacting objects. Similar to other generative approaches, RELATE is trained end-to-end on raw, unlabeled data. RELATE combines an object-centric GAN formulation with a model that explicitly accounts for correlations between individual objects. This allows the model to generate realistic scenes and videos from a physically-interpretable parameterization. Furthermore, we show that modeling the object correlation is necessary to learn to disentangle object positions and identity. We find that RELATE is also amenable to physically realistic scene editing and that it significantly outperforms prior art in object-centric scene generation in both synthetic (CLEVR, ShapeStacks) and real-world data (cars). In addition, in contrast to state-of-the-art methods in object-centric generative modeling, RELATE also extends naturally to dynamic scenes and generates videos of high visual fidelity. Source code, datasets and more results are available at http://geometry.cs.ucl.ac.uk/projects/2020/relate/.

References in corpus (18)

• Auto-Encoding Variational Bayes
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• Interaction Networks for Learning about Objects, Relations and Physics
• Multi-Object Representation Learning with Iterative Variational Inference
• On Convergence and Stability of GANs
• MONet: Unsupervised Scene Decomposition and Representation
• Towards Accurate Generative Models of Video: A New Metric & Challenges
• A Compositional Object-Based Approach to Learning Physical Dynamics
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
• BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
• Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
• Reasoning About Physical Interactions with Object-Oriented Prediction and Planning
• Investigating Object Compositionality in Generative Adversarial Networks
• Object-Centric Image Generation with Factored Depths, Locations, and Appearances
• Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis
• End-to-end Recurrent Multi-Object Tracking and Trajectory Prediction with Relational Reasoning

Cited by in corpus (7)

• Generative Adversarial Transformers
• GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
• Compositional Transformers for Scene Generation
• GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving
• Benchmarking Unsupervised Object Representations for Video Sequences
• Unsupervised Object Learning via Common Fate
• Illiterate DALL-E Learns to Compose