Dynamic Horizon Value Estimation for Model-based Reinforcement Learning
arXiv:2009.09593
Abstract
Existing model-based value expansion methods typically leverage a world model for value estimation with a fixed rollout horizon to assist policy learning. However, the fixed rollout with an inaccurate model has a potential to harm the learning process. In this paper, we investigate the idea of using the model knowledge for value expansion adaptively. We propose a novel method called Dynamic-horizon Model-based Value Expansion (DMVE) to adjust the world model usage with different rollout horizons. Inspired by reconstruction-based techniques that can be applied for visual data novelty detection, we utilize a world model with a reconstruction module for image feature extraction, in order to acquire more precise value estimation. The raw and the reconstructed images are both used to determine the appropriate horizon for adaptive value expansion. On several benchmark visual control tasks, experimental results show that DMVE outperforms all baselines in sample efficiency and final performance, indicating that DMVE can achieve more effective and accurate value estimation than state-of-the-art model-based methods.
9 pages, 6 figures
References in corpus (12)
- Continuous control with deep reinforcement learning
- Recurrent World Models Facilitate Policy Evolution
- Learning Latent Dynamics for Planning from Pixels
- Imagination-Augmented Agents for Deep Reinforcement Learning
- dm_control: Software and Tasks for Continuous Control
- Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
- Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
- SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
- Model-Based Reinforcement Learning via Meta-Policy Optimization
- Improving Reconstruction Autoencoder Out-of-distribution Detection with Mahalanobis Distance
- Self-Correcting Models for Model-Based Reinforcement Learning
- Selective Dyna-style Planning Under Limited Model Capacity