Unsupervised State Representation Learning in Atari

Abstract

State representation learning, or the ability to capture latent generative factors of an environment, is crucial for building intelligent agents that can perform a wide variety of tasks. Learning such representations without supervision from rewards is a challenging open problem. We introduce a method that learns state representations by maximizing mutual information across spatially and temporally distinct features of a neural encoder of the observations. We also introduce a new benchmark based on Atari 2600 games where we evaluate representations based on how well they capture the ground truth state variables. We believe this new framework for evaluating representation learning models will be crucial for future representation learning research. Finally, we compare our technique with other state-of-the-art generative and contrastive representation learning methods. The code associated with this work is available at https://github.com/mila-iqia/atari-representation-learning

arXiv commentNeurIPS 2019; v6 fixes a broken figure reference

References in corpus (31)

• Auto-Encoding Variational Bayes
• Representation Learning with Contrastive Predictive Coding
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Playing Atari with Deep Reinforcement Learning
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Learning deep representations by mutual information estimation and maximization
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Learning Representations by Maximizing Mutual Information Across Views
• Adversarially Learned Inference
• Recurrent World Models Facilitate Policy Evolution
• State Representation Learning for Control: An Overview
• Towards a Definition of Disentangled Representations
• Multi-Object Representation Learning with Iterative Variational Inference
• MONet: Unsupervised Scene Decomposition and Representation
• On Variational Bounds of Mutual Information
• Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Understanding intermediate layers using linear classifier probes
• Revisiting Self-Supervised Visual Representation Learning
• Few-Shot Learning Through an Information Retrieval Lens
• Formal Limitations on the Measurement of Mutual Information
• Contingency-Aware Exploration in Reinforcement Learning
• Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
• Independently Controllable Factors
• PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations
• Natural Environment Benchmarks for Reinforcement Learning
• EMI: Exploration with Mutual Information
• Predictive Information
• Object-Oriented Dynamics Predictor
• Wasserstein Dependency Measure for Representation Learning
• Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency

Cited by in corpus (4)

• The TrojAI Software Framework: An OpenSource tool for Embedding Trojans into Deep Learning Models
• Multimodal Information Bottleneck for Deep Reinforcement Learning with Multiple Sensors
• Learning One Representation to Optimize All Rewards
• Transfer Learning of fMRI Dynamics