Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning
arXiv:2010.12142
Abstract
Sample efficiency has been one of the major challenges for deep reinforcement learning. Recently, model-based reinforcement learning has been proposed to address this challenge by performing planning on imaginary trajectories with a learned world model. However, world model learning may suffer from overfitting to training trajectories, and thus model-based value estimation and policy search will be pone to be sucked in an inferior local policy. In this paper, we propose a novel model-based reinforcement learning algorithm, called BrIdging Reality and Dream (BIRD). It maximizes the mutual information between imaginary and real trajectories so that the policy improvement learned from imaginary trajectories can be easily generalized to real trajectories. We demonstrate that our approach improves sample efficiency of model-based planning, and achieves state-of-the-art performance on challenging visual control benchmarks.
arXiv commentPublished on 34th Conference on Neural Information Processing Systems (NeurIPS 2020)
References in corpus (25)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Learning deep representations by mutual information estimation and maximization
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- DeepMind Control Suite
- Reinforcement Learning with Deep Energy-Based Policies
- Model-Based Reinforcement Learning for Atari
- Learning Continuous Control Policies by Stochastic Value Gradients
- Learning Latent Dynamics for Planning from Pixels
- Distributed Distributional Deterministic Policy Gradients
- On Mutual Information Maximization for Representation Learning
- Model-Ensemble Trust-Region Policy Optimization
- Benchmarking Model-Based Reinforcement Learning
- Bridging the Gap Between Value and Policy Based Reinforcement Learning
- Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
- Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
- Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
- Universal Planning Networks
- Neural Predictive Belief Representations
- Model-Based Planning with Discrete and Continuous Actions
- Reinforcement Learning by Value Gradients
- Information-Directed Exploration for Deep Reinforcement Learning
- Regularizing Trajectory Optimization with Denoising Autoencoders
- Learning Representations in Reinforcement Learning:An Information Bottleneck Approach