MONet: Unsupervised Scene Decomposition and Representation

Abstract

The ability to decompose scenes in terms of abstract building blocks is crucial for general intelligence. Where those basic building blocks share meaningful properties, interactions and other regularities across scenes, such decompositions can simplify reasoning and facilitate imagination of novel scenarios. In particular, representing perceptual observations in terms of entities should improve data efficiency and transfer performance on a wide range of tasks. Thus we need models capable of discovering useful decompositions of scenes by identifying units with such regularities and representing them in a common format. To address this problem, we have developed the Multi-Object Network (MONet). In this model, a VAE is trained end-to-end together with a recurrent attention network -- in a purely unsupervised manner -- to provide attention masks around, and reconstructions of, regions of images. We show that this model is capable of learning to decompose and represent challenging 3D scenes into semantically meaningful components, such as objects and background elements.

References in corpus (9)

• Auto-Encoding Variational Bayes
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Deep Convolutional Inverse Graphics Network
• Disentangling by Factorising
• Understanding disentangling in $\beta$-VAE
• Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
• Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs
• Adversarial Scene Editing: Automatic Object Removal from Weak Supervision

Cited by in corpus (113)

• The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
• Multi-Object Representation Learning with Iterative Variational Inference
• Object-Centric Learning with Slot Attention
• Unsupervised deep learning identifies semantic disentanglement in single inferotemporal neurons
• GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
• Localizing Objects with Self-Supervised Transformers and no Labels
• Unsupervised Learning of Object Keypoints for Perception and Control
• SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition
• Contrastive Learning of Structured World Models
• CLEVRER: CoLlision Events for Video REpresentation and Reasoning
• COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
• Unsupervised Object Segmentation by Redrawing
• Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration
• On the Binding Problem in Artificial Neural Networks
• Learning Physical Graph Representations from Visual Scenes
• Object Discovery with a Copy-Pasting GAN
• Unsupervised Model Selection for Variational Disentangled Representation Learning
• Stacked Capsule Autoencoders
• RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces
• From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence
• Entity Abstraction in Visual Model-Based Reinforcement Learning
• Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation
• Unsupervised State Representation Learning in Atari
• Learning 3D Dynamic Scene Representations for Robot Manipulation
• Towards causal generative scene models via competition of experts
• Unsupervised Discovery of Object Radiance Fields
• R-SQAIR: Relational Sequential Attend, Infer, Repeat
• Structured Object-Aware Physics Prediction for Video Modeling and Planning
• Unsupervised part representation by Flow Capsules
• Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding
• Generative Adversarial Transformers
• Emergence of Object Segmentation in Perturbed Generative Models
• GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
• Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs
• Physion: Evaluating Physical Prediction from Vision in Humans and Machines
• Visual Compositional Learning for Human-Object Interaction Detection
• Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video
• Counterfactual Data Augmentation using Locally Factored Dynamics
• Compositional Transformers for Scene Generation
• Unsupervised Learning of Compositional Energy Concepts
• Unsupervised Discovery of 3D Physical Objects from Video
• Variable-rate discrete representation learning
• SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition
• Generative Neurosymbolic Machines
• Improving Generative Imagination in Object-Centric World Models
• If deep learning is the answer, then what is the question?
• AlignNet: Unsupervised Entity Alignment
• HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving
• Unsupervised Video Decomposition using Spatio-temporal Iterative Inference
• Occlusion resistant learning of intuitive physics from videos
• Attention over learned object embeddings enables complex visual reasoning
• Forward Prediction for Physical Reasoning
• Object-Centric Image Generation with Factored Depths, Locations, and Appearances
• What is Wrong with One-Class Anomaly Detection?
• Reconstruction Bottlenecks in Object-Centric Generative Models
• Multi-objects Generation with Amortized Structural Regularization
• A Neural Rendering Framework for Free-Viewpoint Relighting
• Representation learning for improved interpretability and classification accuracy of clinical factors from EEG
• Slot Contrastive Networks: A Contrastive Approach for Representing Objects
• Generalization and Robustness Implications in Object-Centric Learning
• Refactoring Policy for Compositional Generalizability using Self-Supervised Object Proposals
• Product Kanerva Machines: Factorized Bayesian Memory
• Self-supervised Video Object Segmentation by Motion Grouping
• Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis
• Learning Object-Centric Video Models by Contrasting Sets
• Benchmarking Unsupervised Object Representations for Video Sequences
• Levels of Analysis for Machine Learning
• Semantic Compositional Learning for Low-shot Scene Graph Generation
• Imagine That! Leveraging Emergent Affordances for 3D Tool Synthesis
• KANDINSKYPatterns -- An experimental exploration environment for Pattern Analysis and Machine Intelligence
• Unsupervised Object Segmentation with Explicit Localization Module
• Better Set Representations For Relational Reasoning
• Unsupervised object-centric video generation and decomposition in 3D
• Boosting Object Representation Learning via Motion and Object Continuity
• Deep Sets for Generalization in RL
• Disentangling causal effects for hierarchical reinforcement learning
• Unsupervised Object Keypoint Learning using Local Spatial Predictability
• From Language Games to Drawing Games
• Visually Grounded Compound PCFGs
• Discovering Dynamic Salient Regions for Spatio-Temporal Graph Neural Networks
• LAVAE: Disentangling Location and Appearance
• SLASH: Embracing Probabilistic Circuits into Neural Answer Set Programming
• Grounding Spatio-Temporal Language with Transformers
• Disentangled Relational Representations for Explaining and Learning from Demonstration
• Self-supervised Visual Reinforcement Learning with Object-centric Representations
• Control What You Can: Intrinsically Motivated Task-Planning Agent
• Unsupervised Part Discovery from Contrastive Reconstruction
• Pretrained Encoders are All You Need
• Causal World Models by Unsupervised Deconfounding of Physical Dynamics
• Instance Separation Emerges from Inpainting
• Learning to Manipulate Individual Objects in an Image
• Generalization to Novel Objects using Prior Relational Knowledge
• Language-Mediated, Object-Centric Representation Learning
• Knowledge-Guided Object Discovery with Acquired Deep Impressions
• Jointly Learning Truth-Conditional Denotations and Groundings using Parallel Attention
• pix2rule: End-to-end Neuro-symbolic Rule Learning
• Self-supervised Reinforcement Learning with Independently Controllable Subgoals
• Unsupervised Object Learning via Common Fate
• GMAIR: Unsupervised Object Detection Based on Spatial Attention and Gaussian Mixture
• PatchNet: Unsupervised Object Discovery based on Patch Embedding
• Image-Graph-Image Translation via Auto-Encoding
• Polyline Generative Navigable Space Segmentation for Autonomous Visual Navigation
• Disassembling Object Representations without Labels
• Illiterate DALL-E Learns to Compose
• Constellation: Learning relational abstractions over objects for compositional imagination
• Attribution Mask: Filtering Out Irrelevant Features By Recursively Focusing Attention on Inputs of DNNs
• MarioNette: Self-Supervised Sprite Learning
• Neuro-Symbolic Forward Reasoning
• Recurrent Attention Models with Object-centric Capsule Representation for Multi-object Recognition
• Group Equivariant Subsampling
• Understanding in Artificial Intelligence
• Conditional Object-Centric Learning from Video
• The Impact of Negative Sampling on Contrastive Structured World Models