Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitation

Abstract

Model-based deep reinforcement learning has achieved success in various domains that require high sample efficiencies, such as Go and robotics. However, there are some remaining issues, such as planning efficient explorations to learn more accurate dynamic models, evaluating the uncertainty of the learned models, and more rational utilization of models. To mitigate these issues, we present MEEE, a model-ensemble method that consists of optimistic exploration and weighted exploitation. During exploration, unlike prior methods directly selecting the optimal action that maximizes the expected accumulative return, our agent first generates a set of action candidates and then seeks out the optimal action that takes both expected return and future observation novelty into account. During exploitation, different discounted weights are assigned to imagined transition tuples according to their model uncertainty respectively, which will prevent model predictive error propagation in agent training. Experiments on several challenging continuous control benchmark tasks demonstrated that our approach outperforms other model-free and model-based state-of-the-art methods, especially in sample complexity.

arXiv comment7 pages, 5 figures, accepted by IEEE International Conference on Robotics and Automation 2021 (IEEE ICRA 2021)

References in corpus (15)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Variational Dropout and the Local Reparameterization Trick
• Model-Based Reinforcement Learning for Atari
• Large-Scale Study of Curiosity-Driven Learning
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Model-Ensemble Trust-Region Policy Optimization
• Agent57: Outperforming the Atari Human Benchmark
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Model-Based Active Exploration
• Planning to Explore via Self-Supervised World Models
• Learning to Perform Physics Experiments via Deep Reinforcement Learning
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• RLCard: A Toolkit for Reinforcement Learning in Card Games
• Combating the Compounding-Error Problem with a Multi-step Model
• Self-Supervised Exploration via Disagreement

Cited by in corpus (1)

• Robust Model-based Reinforcement Learning for Autonomous Greenhouse Control