SCALOR: Generative World Models with Scalable Object Representations

Abstract

Scalability in terms of object density in a scene is a primary challenge in unsupervised sequential object-oriented representation learning. Most of the previous models have been shown to work only on scenes with a few objects. In this paper, we propose SCALOR, a probabilistic generative world model for learning SCALable Object-oriented Representation of a video. With the proposed spatially-parallel attention and proposal-rejection mechanisms, SCALOR can deal with orders of magnitude larger numbers of objects compared to the previous state-of-the-art models. Additionally, we introduce a background module that allows SCALOR to model complex dynamic backgrounds as well as many foreground objects in the scene. We demonstrate that SCALOR can deal with crowded scenes containing up to a hundred objects while jointly modeling complex dynamic backgrounds. Importantly, SCALOR is the first unsupervised object representation model shown to work for natural scenes containing several tens of moving objects.

First two authors contributed equally. Published at ICLR 2020

Cited by in corpus (20)

• Object-Centric Learning with Slot Attention
• On the Binding Problem in Artificial Neural Networks
• From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence
• GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
• Generative Neurosymbolic Machines
• Unsupervised Video Decomposition using Spatio-temporal Iterative Inference
• Reconstruction Bottlenecks in Object-Centric Generative Models
• End-to-end Recurrent Multi-Object Tracking and Trajectory Prediction with Relational Reasoning
• Generalization and Robustness Implications in Object-Centric Learning
• Unsupervised object-centric video generation and decomposition in 3D
• Benchmarking Unsupervised Object Representations for Video Sequences
• Unsupervised Part Discovery from Contrastive Reconstruction
• Jointly Learning Truth-Conditional Denotations and Groundings using Parallel Attention
• Self-Supervised Decomposition, Disentanglement and Prediction of Video Sequences while Interpreting Dynamics: A Koopman Perspective
• Self-supervised Reinforcement Learning with Independently Controllable Subgoals
• Illiterate DALL-E Learns to Compose
• Opening up Open-World Tracking
• MarioNette: Self-Supervised Sprite Learning
• Conditional Object-Centric Learning from Video
• Unsupervised Video Prediction from a Single Frame by Estimating 3D Dynamic Scene Structure