Contrastive Learning of Structured World Models

Abstract

A structured understanding of our world in terms of objects, relations, and hierarchies is an important component of human cognition. Learning such a structured world model from raw sensory data remains a challenge. As a step towards this goal, we introduce Contrastively-trained Structured World Models (C-SWMs). C-SWMs utilize a contrastive approach for representation learning in environments with compositional structure. We structure each state embedding as a set of object representations and their relations, modeled by a graph neural network. This allows objects to be discovered from raw pixel observations without direct supervision as part of the learning process. We evaluate C-SWMs on compositional environments involving multiple interacting objects that can be manipulated independently by an agent, simple Atari games, and a multi-object physics simulation. Our experiments demonstrate that C-SWMs can overcome limitations of models based on pixel reconstruction and outperform typical representatives of this model class in highly structured environments, while learning interpretable object-based representations.

ICLR 2020

References in corpus (24)

• Adam: A Method for Stochastic Optimization
• Semi-Supervised Classification with Graph Convolutional Networks
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Empirical Evaluation of Rectified Activations in Convolutional Network
• A Review of Relational Machine Learning for Knowledge Graphs
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Gated Graph Sequence Neural Networks
• Model-Based Reinforcement Learning for Atari
• Learning Latent Dynamics for Planning from Pixels
• Graph networks as learnable physics engines for inference and control
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
• Multi-Object Representation Learning with Iterative Variational Inference
• Learning Video Representations using Contrastive Bidirectional Transformer
• MONet: Unsupervised Scene Decomposition and Representation
• Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
• Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
• Learning Plannable Representations with Causal InfoGAN
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
• Curiosity Driven Exploration of Learned Disentangled Goal Spaces
• Efficient Model-Based Deep Reinforcement Learning with Variational State Tabulation
• Disentangling the independently controllable factors of variation by interacting with the world
• Learning Robotic Manipulation through Visual Planning and Acting
• Relational Action Forecasting

Cited by in corpus (22)

• Contrastive Representation Learning: A Framework and Review
• Graph Self-Supervised Learning: A Survey
• Decoupling Representation Learning from Reinforcement Learning
• Contrastive Knowledge Graph Error Detection
• Multi-Label Contrastive Learning for Abstract Visual Reasoning
• Discrete and continuous representations and processing in deep learning: Looking forward
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Offline Reinforcement Learning from Images with Latent Space Models
• CAPT: Contrastive Pre-Training for Learning Denoised Sequence Representations
• Discrete-Valued Neural Communication
• Counterfactual Data Augmentation using Locally Factored Dynamics
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• CausalCity: Complex Simulations with Agency for Causal Discovery and Reasoning
• Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning
• Slot Contrastive Networks: A Contrastive Approach for Representing Objects
• Learning Object-Centric Video Models by Contrasting Sets
• Function Contrastive Learning of Transferable Meta-Representations
• KANDINSKYPatterns -- An experimental exploration environment for Pattern Analysis and Machine Intelligence
• Low-Dimensional State and Action Representation Learning with MDP Homomorphism Metrics
• Disentangled (Un)Controllable Features
• Reasoning-Modulated Representations
• Reinforcement Learning for Sparse-Reward Object-Interaction Tasks in a First-person Simulated 3D Environment