Efficient Model-Based Deep Reinforcement Learning with Variational State Tabulation

Abstract

Modern reinforcement learning algorithms reach super-human performance on many board and video games, but they are sample inefficient, i.e. they typically require significantly more playing experience than humans to reach an equal performance level. To improve sample efficiency, an agent may build a model of the environment and use planning methods to update its policy. In this article we introduce Variational State Tabulation (VaST), which maps an environment with a high-dimensional state space (e.g. the space of visual inputs) to an abstract tabular model. Prioritized sweeping with small backups, a highly efficient planning method, can then be used to update state-action values. We show how VaST can rapidly learn to maximize reward in tasks like 3D navigation and efficiently adapt to sudden changes in rewards or transition probabilities.

arXiv commentAccepted at ICML 2018; camera-ready version

References in corpus (3)

• Adam: A Method for Stochastic Optimization
• Prioritized Experience Replay
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables

Cited by in corpus (16)

• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• Learning Plannable Representations with Causal InfoGAN
• Contrastive Learning of Structured World Models
• Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning
• World Model as a Graph: Learning Latent Landmarks for Planning
• The Value Equivalence Principle for Model-Based Reinforcement Learning
• A Sufficient Statistic for Influence in Structured Multiagent Environments
• Learning Markov State Abstractions for Deep Reinforcement Learning
• DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPs
• Agent Modelling under Partial Observability for Deep Reinforcement Learning
• Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning
• Skill Transfer in Deep Reinforcement Learning under Morphological Heterogeneity
• Tutorial and Survey on Probabilistic Graphical Model and Variational Inference in Deep Reinforcement Learning
• Hierarchical Representation Learning for Markov Decision Processes
• Variational State-Space Models for Localisation and Dense 3D Mapping in 6 DoF
• Proper Value Equivalence