Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions

Abstract

Common-sense physical reasoning is an essential ingredient for any intelligent agent operating in the real-world. For example, it can be used to simulate the environment, or to infer the state of parts of the world that are currently unobserved. In order to match real-world conditions this causal knowledge must be learned without access to supervised data. To address this problem we present a novel method that learns to discover objects and model their physical interactions from raw visual images in a purely \emph{unsupervised} fashion. It incorporates prior knowledge about the compositional nature of human perception to factor interactions between object-pairs and learn efficiently. On videos of bouncing balls we show the superior modelling capabilities of our method compared to other unsupervised neural approaches that do not incorporate such prior knowledge. We demonstrate its ability to handle occlusion and show that it can extrapolate learned knowledge to scenes with different numbers of objects.

arXiv commentAccepted to ICLR 2018

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Neural Machine Translation by Jointly Learning to Align and Translate
• Playing Atari with Deep Reinforcement Learning
• Neural Message Passing for Quantum Chemistry
• DRAW: A Recurrent Neural Network For Image Generation
• A simple neural network module for relational reasoning
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
• Neural Programmer-Interpreters
• Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• Learning Visual Predictive Models of Physics for Playing Billiards
• Making Neural Programming Architectures Generalize via Recursion
• Programming with a Differentiable Forth Interpreter
• Visual Interaction Networks
• Learning to Infer Graphics Programs from Hand-Drawn Images
• On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models

Cited by in corpus (47)

• A Review on Deep Learning Techniques for Video Prediction
• MONet: Unsupervised Scene Decomposition and Representation
• GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
• Learning to Decompose and Disentangle Representations for Video Prediction
• Object Level Visual Reasoning in Videos
• SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition
• Recurrent Independent Mechanisms
• Contrastive Learning of Structured World Models
• Situation-Aware Pedestrian Trajectory Prediction with Spatio-Temporal Attention Model
• Is Attention Better Than Matrix Decomposition?
• Deep Iterative and Adaptive Learning for Graph Neural Networks
• Investigating Object Compositionality in Generative Adversarial Networks
• Entity Abstraction in Visual Model-Based Reinforcement Learning
• Graph Element Networks: adaptive, structured computation and memory
• Towards causal generative scene models via competition of experts
• Unsupervised part representation by Flow Capsules
• Coordination Among Neural Modules Through a Shared Global Workspace
• GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
• Object Files and Schemata: Factorizing Declarative and Procedural Knowledge in Dynamical Systems
• Discovering Nonlinear Relations with Minimum Predictive Information Regularization
• Factorised Neural Relational Inference for Multi-Interaction Systems
• Discrete-Valued Neural Communication
• Unsupervised Learning of Compositional Energy Concepts
• Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction
• Dynamic Inference with Neural Interpreters
• Generative Neurosymbolic Machines
• Unsupervised Video Decomposition using Spatio-temporal Iterative Inference
• Forward Prediction for Physical Reasoning
• Reconstruction Bottlenecks in Object-Centric Generative Models
• Generalization and Robustness Implications in Object-Centric Learning
• Slot Contrastive Networks: A Contrastive Approach for Representing Objects
• Learning Object-Centric Video Models by Contrasting Sets
• Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning
• SENSORIMOTOR GRAPH: Action-Conditioned Graph Neural Network for Learning Robotic Soft Hand Dynamics
• Graph Neural Processes: Towards Bayesian Graph Neural Networks
• Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning
• Uncovering Closed-form Governing Equations of Nonlinear Dynamics from Videos
• LAVAE: Disentangling Location and Appearance
• Alternating ConvLSTM: Learning Force Propagation with Alternate State Updates
• Neural Allocentric Intuitive Physics Prediction from Real Videos
• Mega-Reward: Achieving Human-Level Play without Extrinsic Rewards
• Causal World Models by Unsupervised Deconfounding of Physical Dynamics
• Modular meta-learning in abstract graph networks for combinatorial generalization
• Cut-Based Graph Learning Networks to Discover Compositional Structure of Sequential Video Data
• Reasoning-Modulated Representations
• Discovering Latent Representations of Relations for Interacting Systems
• A Neural-Symbolic Framework for Mental Simulation