Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL
arXiv:2106.15883
Abstract
Despite a series of recent successes in reinforcement learning (RL), many RL algorithms remain sensitive to hyperparameters. As such, there has recently been interest in the field of AutoRL, which seeks to automate design decisions to create more general algorithms. Recent work suggests that population based approaches may be effective AutoRL algorithms, by learning hyperparameter schedules on the fly. In particular, the PB2 algorithm is able to achieve strong performance in RL tasks by formulating online hyperparameter optimization as time varying GP-bandit problem, while also providing theoretical guarantees. However, PB2 is only designed to work for continuous hyperparameters, which severely limits its utility in practice. In this paper we introduce a new (provably) efficient hierarchical approach for optimizing both continuous and categorical variables, using a new time-varying bandit algorithm specifically designed for the population based training regime. We evaluate our approach on the challenging Procgen benchmark, where we show that explicitly modelling dependence between data augmentation and other hyperparameters improves generalization.
References in corpus (23)
- Practical Bayesian Optimization of Machine Learning Algorithms
- Playing Atari with Deep Reinforcement Learning
- Trust Region Policy Optimization
- A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
- Dota 2 with Large Scale Deep Reinforcement Learning
- Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
- Deep Reinforcement Learning that Matters
- Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
- Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules
- Assessing Generalization in Deep Reinforcement Learning
- Emergent Coordination Through Competition
- Gotta Learn Fast: A New Benchmark for Generalization in RL
- Bayesian Optimization in AlphaGo
- Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
- Kickstarting Deep Reinforcement Learning
- Time-Varying Gaussian Process Bandit Optimization
- Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment
- Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research
- Asynchronous Batch Bayesian Optimisation with Improved Local Penalisation
- Evolving Reinforcement Learning Algorithms
- Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels
- Fast Efficient Hyperparameter Tuning for Policy Gradients
- An Empirical Study on Hyperparameters and their Interdependence for RL Generalization