On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning

Abstract

Model-based Reinforcement Learning (MBRL) is a promising framework for learning control in a data-efficient manner. MBRL algorithms can be fairly complex due to the separate dynamics modeling and the subsequent planning algorithm, and as a result, they often possess tens of hyperparameters and architectural choices. For this reason, MBRL typically requires significant human expertise before it can be applied to new problems and domains. To alleviate this problem, we propose to use automatic hyperparameter optimization (HPO). We demonstrate that this problem can be tackled effectively with automated HPO, which we demonstrate to yield significantly improved performance compared to human experts. In addition, we show that tuning of several MBRL hyperparameters dynamically, i.e. during the training itself, further improves the performance compared to using static hyperparameters which are kept fixed for the whole training. Finally, our experiments provide valuable insights into the effects of several hyperparameters, such as plan horizon or learning rate and their influence on the stability of training and resulting rewards.

19 pages, accepted by AISTATS 2021

References in corpus (3)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• Ctrl-Z: Recovering from Instability in Reinforcement Learning

Cited by in corpus (9)

• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Survival Prediction of Children Undergoing Hematopoietic Stem Cell Transplantation Using Different Machine Learning Classifiers by Performing Chi-squared Test and Hyper-parameter Optimization: A Retrospective Analysis
• Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML
• Forward and inverse reinforcement learning sharing network weights and hyperparameters
• Safe Learning MPC with Limited Model Knowledge and Data
• On Effective Scheduling of Model-based Reinforcement Learning
• MDP Playground: An Analysis and Debug Testbed for Reinforcement Learning
• Learning Off-Policy with Online Planning
• On-Policy Model Errors in Reinforcement Learning