Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey
arXiv:2008.05598
Abstract
Deep reinforcement learning has shown remarkable success in the past few years. Highly complex sequential decision making problems have been solved in tasks such as game playing and robotics. Unfortunately, the sample complexity of most deep reinforcement learning methods is high, precluding their use in some important applications. Model-based reinforcement learning creates an explicit model of the environment dynamics to reduce the need for environment samples. Current deep learning methods use high-capacity networks to solve high-dimensional problems. Unfortunately, high-capacity models typically require many samples, negating the potential benefit of lower sample complexity in model-based methods. A challenge for deep model-based methods is therefore to achieve high predictive power while maintaining low sample complexity. In recent years, many model-based methods have been introduced to address this challenge. In this paper, we survey the contemporary model-based landscape. First we discuss definitions and relations to other fields. We propose a taxonomy based on three approaches: using explicit planning on given transitions, using explicit planning on learned transitions, and end-to-end learning of both planning and transitions. We use these approaches to organize a comprehensive overview of important recent developments such as latent models. We describe methods and benchmarks, and we suggest directions for future work for each of the approaches. Among promising research directions are curriculum learning, uncertainty modeling, and use of latent models for transfer learning.
References in corpus (35)
- Auto-Encoding Variational Bayes
- Continuous control with deep reinforcement learning
- Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
- Playing Atari with Deep Reinforcement Learning
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- Dota 2 with Large Scale Deep Reinforcement Learning
- Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
- Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
- StarCraft II: A New Challenge for Reinforcement Learning
- Recurrent World Models Facilitate Policy Evolution
- Model-Based Reinforcement Learning for Atari
- Learning Continuous Control Policies by Stochastic Value Gradients
- Learning Latent Dynamics for Planning from Pixels
- Benchmarking Model-Based Reinforcement Learning
- Thinking Fast and Slow with Deep Learning and Tree Search
- Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
- Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control
- Relational Deep Reinforcement Learning
- The Predictron: End-To-End Learning and Planning
- Recurrent Environment Simulators
- Universal Planning Networks
- Emergent Coordination Through Competition
- Learning model-based planning from scratch
- Planning to Explore via Self-Supervised World Models
- Learning and Querying Fast Generative Models for Reinforcement Learning
- Metacontrol for Adaptive Imagination-Based Optimization
- Ranked Reward: Enabling Self-Play Reinforcement Learning for Combinatorial Optimization
- Generalized Value Iteration Networks: Life Beyond Lattices
- A0C: Alpha Zero in Continuous Action Space
- Value Iteration Networks on Multiple Levels of Abstraction
- Policy Gradient Search: Online Planning and Expert Iteration without Search Trees
- Learning to Play No-Press Diplomacy with Best Response Policy Iteration
- Value Propagation Networks
- Surprising Negative Results for Generative Adversarial Tree Search
- The Second Type of Uncertainty in Monte Carlo Tree Search