DeepMDP: Learning Continuous Latent Space Models for Representation Learning

Abstract

Many reinforcement learning (RL) tasks provide the agent with high-dimensional observations that can be simplified into low-dimensional continuous states. To formalize this process, we introduce the concept of a DeepMDP, a parameterized latent space model that is trained via the minimization of two tractable losses: prediction of rewards and prediction of the distribution over next latent states. We show that the optimization of these objectives guarantees (1) the quality of the latent space as a representation of the state space and (2) the quality of the DeepMDP as a model of the environment. We connect these results to prior work in the bisimulation literature, and explore the use of a variety of metrics. Our theoretical findings are substantiated by the experimental result that a trained DeepMDP recovers the latent structure underlying high-dimensional observations on a synthetic environment. Finally, we show that learning a DeepMDP as an auxiliary task in the Atari 2600 domain leads to large performance improvements over model-free RL.

13 pages main text, 16 pages appendix. ICML 2019

References in corpus (18)

• Equivalence of distance-based and RKHS-based statistics in hypothesis testing
• Recurrent World Models Facilitate Policy Evolution
• Model-Based Reinforcement Learning for Atari
• Learning to Navigate in Complex Environments
• The Cramer Distance as a Solution to Biased Wasserstein Gradients
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• The Predictron: End-To-End Learning and Planning
• Distributional Reinforcement Learning with Quantile Regression
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Visual Interaction Networks
• Learning and Querying Fast Generative Models for Reinforcement Learning
• Hyperbolic Discounting and Learning over Multiple Horizons
• Lipschitz Continuity in Model-based Reinforcement Learning
• A Comparative Analysis of Expected and Distributional Reinforcement Learning
• Probabilistic Recurrent State-Space Models

Cited by in corpus (32)

• Dream to Control: Learning Behaviors by Latent Imagination
• Contrastive Learning of Structured World Models
• Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
• Invariant Causal Prediction for Block MDPs
• A Geometric Perspective on Optimal Representations for Reinforcement Learning
• Discriminative Particle Filter Reinforcement Learning for Complex Partial Observations
• Predictive Information Accelerates Learning in RL
• Objective Mismatch in Model-based Reinforcement Learning
• Return-Based Contrastive Representation Learning for Reinforcement Learning
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Offline Reinforcement Learning from Images with Latent Space Models
• Knowledge-integrated AutoEncoder Model
• The Value Equivalence Principle for Model-Based Reinforcement Learning
• How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimization
• A Sufficient Statistic for Influence in Structured Multiagent Environments
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• Revisit Recommender System in the Permutation Prospective
• Novelty Search in Representational Space for Sample Efficient Exploration
• The Differentiable Cross-Entropy Method
• Weakly-Supervised Reinforcement Learning for Controllable Behavior
• Learning Robust State Abstractions for Hidden-Parameter Block MDPs
• GRN: Generative Rerank Network for Context-wise Recommendation
• Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning
• Value-driven Hindsight Modelling
• ExTra: Transfer-guided Exploration
• Low-Dimensional State and Action Representation Learning with MDP Homomorphism Metrics
• Learning Representations for Pixel-based Control: What Matters and Why?
• Provable RL with Exogenous Distractors via Multistep Inverse Dynamics
• Towards Robust Bisimulation Metric Learning
• A Geometric Perspective on Self-Supervised Policy Adaptation
• Low-Variance Policy Gradient Estimation with World Models
• A Self-Supervised Auxiliary Loss for Deep RL in Partially Observable Settings