Multi-Object Representation Learning with Iterative Variational Inference

Abstract

Human perception is structured around objects which form the basis for our higher-level cognition and impressive systematic generalization abilities. Yet most work on representation learning focuses on feature learning without even considering multiple objects, or treats segmentation as an (often supervised) preprocessing step. Instead, we argue for the importance of learning to segment and represent objects jointly. We demonstrate that, starting from the simple assumption that a scene is composed of multiple entities, it is possible to learn to segment images into interpretable objects with disentangled representations. Our method learns -- without supervision -- to inpaint occluded parts, and extrapolates to scenes with more objects and to unseen objects with novel feature combinations. We also show that, due to the use of iterative variational inference, our system is able to learn multi-modal posteriors for ambiguous inputs and extends naturally to sequences.

References in corpus (16)

• Natural Scales in Geographical Patterns
• YOLOv3: An Incremental Improvement
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• DRAW: A Recurrent Neural Network For Image Generation
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Disentangling by Factorising
• Towards a Definition of Disentangled Representations
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
• MONet: Unsupervised Scene Decomposition and Representation
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Learning Deep Disentangled Embeddings with the F-Statistic Loss
• Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs
• Iterative Amortized Inference
• A General Method for Amortizing Variational Filtering

Cited by in corpus (82)

• Object-Centric Learning with Slot Attention
• GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
• Are Disentangled Representations Helpful for Abstract Visual Reasoning?
• Space-Time Correspondence as a Contrastive Random Walk
• Localizing Objects with Self-Supervised Transformers and no Labels
• Unsupervised Learning of Object Keypoints for Perception and Control
• SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition
• Contrastive Learning of Structured World Models
• Deep Set Prediction Networks
• Unsupervised Object Segmentation by Redrawing
• Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration
• Object Discovery with a Copy-Pasting GAN
• Stacked Capsule Autoencoders
• RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces
• Investigating Object Compositionality in Generative Adversarial Networks
• Entity Abstraction in Visual Model-Based Reinforcement Learning
• Unsupervised State Representation Learning in Atari
• Multi-Task Reinforcement Learning with Context-based Representations
• Unsupervised Discovery of Object Radiance Fields
• R-SQAIR: Relational Sequential Attend, Infer, Repeat
• Unsupervised part representation by Flow Capsules
• Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding
• Generative Adversarial Transformers
• Emergence of Object Segmentation in Perturbed Generative Models
• GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
• Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video
• Compositional Transformers for Scene Generation
• Unsupervised Learning of Compositional Energy Concepts
• Unsupervised Discovery of 3D Physical Objects from Video
• Variable-rate discrete representation learning
• Generative Neurosymbolic Machines
• SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition
• Improving Generative Imagination in Object-Centric World Models
• AlignNet: Unsupervised Entity Alignment
• Attention over learned object embeddings enables complex visual reasoning
• Structured Generative Models for Scene Understanding
• Occlusion resistant learning of intuitive physics from videos
• Weakly Supervised Learning of Multi-Object 3D Scene Decompositions Using Deep Shape Priors
• Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction
• Compositional Visual Generation and Inference with Energy Based Models
• Forward Prediction for Physical Reasoning
• Object-Centric Image Generation with Factored Depths, Locations, and Appearances
• Unsupervised Video Decomposition using Spatio-temporal Iterative Inference
• Multi-objects Generation with Amortized Structural Regularization
• Reconstruction Bottlenecks in Object-Centric Generative Models
• Self-supervised Video Object Segmentation by Motion Grouping
• Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis
• Learning Task-General Representations with Generative Neuro-Symbolic Modeling
• Slot Contrastive Networks: A Contrastive Approach for Representing Objects
• Refactoring Policy for Compositional Generalizability using Self-Supervised Object Proposals
• Generalization and Robustness Implications in Object-Centric Learning
• Iterative Amortized Policy Optimization
• Unsupervised object-centric video generation and decomposition in 3D
• Benchmarking Unsupervised Object Representations for Video Sequences
• Better Set Representations For Relational Reasoning
• Imagine That! Leveraging Emergent Affordances for 3D Tool Synthesis
• Levels of Analysis for Machine Learning
• LAVAE: Disentangling Location and Appearance
• Unsupervised Part Discovery from Contrastive Reconstruction
• Deep Sets for Generalization in RL
• SLASH: Embracing Probabilistic Circuits into Neural Answer Set Programming
• Disentangled Relational Representations for Explaining and Learning from Demonstration
• Boosting Object Representation Learning via Motion and Object Continuity
• Grounding Spatio-Temporal Language with Transformers
• Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation
• Generalization to Novel Objects using Prior Relational Knowledge
• Analytic Manifold Learning: Unifying and Evaluating Representations for Continuous Control
• Capsule Networks -- A Probabilistic Perspective
• Language-Mediated, Object-Centric Representation Learning
• Binding and Perspective Taking as Inference in a Generative Neural Network Model
• Jointly Learning Truth-Conditional Denotations and Groundings using Parallel Attention
• Unsupervised Object Learning via Common Fate
• Self-supervised Reinforcement Learning with Independently Controllable Subgoals
• Illiterate DALL-E Learns to Compose
• Relevance-Guided Modeling of Object Dynamics for Reinforcement Learning
• Conditional Object-Centric Learning from Video
• Polyline Generative Navigable Space Segmentation for Autonomous Visual Navigation
• Joint Perception and Control as Inference with an Object-based Implementation
• Understanding in Artificial Intelligence
• Learning to Represent and Predict Sets with Deep Neural Networks
• Boxhead: A Dataset for Learning Hierarchical Representations
• A Proposal for Intelligent Agents with Episodic Memory