Forward and Reverse Gradient-Based Hyperparameter Optimization
arXiv:1703.01785
Abstract
We study two procedures (reverse-mode and forward-mode) for computing the gradient of the validation error with respect to the hyperparameters of any iterative learning algorithm such as stochastic gradient descent. These procedures mirror two methods of computing gradients for recurrent neural networks and have different trade-offs in terms of running time and space requirements. Our formulation of the reverse-mode procedure is linked to previous work by Maclaurin et al. [2015] but does not require reversible dynamics. The forward-mode procedure is suitable for real-time hyperparameter updates, which may significantly speed up hyperparameter optimization on large datasets. We present experiments on data cleaning and on learning task interactions. We also present one large-scale experiment where the use of previous gradient-based methods would be prohibitive.
arXiv comment- Posted the ICML Camera Ready version. - Added a link to a newer package implementation of the algorithms
References in corpus (8)
- Adam: A Method for Stochastic Optimization
- Practical Bayesian Optimization of Machine Learning Algorithms
- Neural Architecture Search with Reinforcement Learning
- Scalable Bayesian Optimization Using Deep Neural Networks
- Gradient-based Hyperparameter Optimization through Reversible Learning
- Hyperparameter optimization with approximate gradient
- Learning What Data to Learn
- Efficient Output Kernel Learning for Multiple Tasks
Cited by in corpus (15)
- Bilevel Programming for Hyperparameter Optimization and Meta-Learning
- Learning the Effect of Registration Hyperparameters with HyperMorph
- A Self-Tuning Actor-Critic Algorithm
- Software Engineering for Fairness: A Case Study with Hyperparameter Optimization
- Guarantees for Tuning the Step Size using a Learning-to-Learn Approach
- Efficient Online Hyperparameter Optimization for Kernel Ridge Regression with Applications to Traffic Time Series Prediction
- Gradient-based algorithms for multi-objective bi-level optimization
- Efficient hyperparameter optimization by way of PAC-Bayes bound minimization
- Efficient Gradient Approximation Method for Constrained Bilevel Optimization
- Hyper-parameter Tuning under a Budget Constraint
- Reducing The Search Space For Hyperparameter Optimization Using Group Sparsity
- Hyperparameter Optimization in Neural Networks via Structured Sparse Recovery
- AutoPruning for Deep Neural Network with Dynamic Channel Masking
- Variationally Inferred Sampling Through a Refined Bound for Probabilistic Programs
- Far-HO: A Bilevel Programming Package for Hyperparameter Optimization and Meta-Learning