Fast Efficient Hyperparameter Tuning for Policy Gradients

Abstract

The performance of policy gradient methods is sensitive to hyperparameter settings that must be tuned for any new application. Widely used grid search methods for tuning hyperparameters are sample inefficient and computationally expensive. More advanced methods like Population Based Training that learn optimal schedules for hyperparameters instead of fixed settings can yield better results, but are also sample inefficient and computationally expensive. In this paper, we propose Hyperparameter Optimisation on the Fly (HOOF), a gradient-free algorithm that requires no more than one training run to automatically adapt the hyperparameter that affect the policy update directly through the gradient. The main idea is to use existing trajectories sampled by the policy gradient method to optimise a one-step improvement objective, yielding a sample and computationally efficient algorithm that is easy to implement. Our experimental results across multiple domains and algorithms show that using HOOF to learn these hyperparameter schedules leads to faster learning with improved performance.

arXiv commentNeurIPS 2019

References in corpus (21)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Practical Bayesian Optimization of Machine Learning Algorithms
• Trust Region Policy Optimization
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Generative Adversarial Imitation Learning
• Deep Reinforcement Learning that Matters
• Hybrid Reward Architecture for Reinforcement Learning
• Meta-Gradient Reinforcement Learning
• Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
• Hyperparameter optimization with approximate gradient
• Freeze-Thaw Bayesian Optimization
• Input Warping for Bayesian Optimization of Non-stationary Functions
• Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
• How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategies
• Bayesian Optimization in AlphaGo
• Parallel Bayesian Global Optimization of Expensive Functions
• Parallel Predictive Entropy Search for Batch Global Optimization of Expensive Objective Functions
• Hyperbolic Discounting and Learning over Multiple Horizons
• Benchmarking Reinforcement Learning Algorithms on Real-World Robots
• TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learning
• Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methods

Cited by in corpus (7)

• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• A Self-Tuning Actor-Critic Algorithm
• Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits
• Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategies
• Towards Automatic Actor-Critic Solutions to Continuous Control
• Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL
• AReN: Assured ReLU NN Architecture for Model Predictive Control of LTI Systems