Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits

Abstract

Many of the recent triumphs in machine learning are dependent on well-tuned hyperparameters. This is particularly prominent in reinforcement learning (RL) where a small change in the configuration can lead to failure. Despite the importance of tuning hyperparameters, it remains expensive and is often done in a naive and laborious way. A recent solution to this problem is Population Based Training (PBT) which updates both weights and hyperparameters in a single training run of a population of agents. PBT has been shown to be particularly effective in RL, leading to widespread use in the field. However, PBT lacks theoretical guarantees since it relies on random heuristics to explore the hyperparameter space. This inefficiency means it typically requires vast computational resources, which is prohibitive for many small and medium sized labs. In this work, we introduce the first provably efficient PBT-style algorithm, Population-Based Bandits (PB2). PB2 uses a probabilistic model to guide the search in an efficient way, making it possible to discover high performing hyperparameter configurations with far fewer agents than typically required by PBT. We show in a series of RL experiments that PB2 is able to achieve high performance with a modest computational budget.

Camera-ready version, NeurIPS 2020

References in corpus (10)

• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Dota 2 with Large Scale Deep Reinforcement Learning
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Tune: A Research Platform for Distributed Model Selection and Training
• Emergent Coordination Through Competition
• Bayesian Optimization in AlphaGo
• Kickstarting Deep Reinforcement Learning
• Asynchronous Batch Bayesian Optimisation with Improved Local Penalisation
• Fast Efficient Hyperparameter Tuning for Policy Gradients

Cited by in corpus (4)

• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• On Controller Tuning with Time-Varying Bayesian Optimization
• Domain Knowledge Injection in Bayesian Search for New Materials