Stabilizing Bi-Level Hyperparameter Optimization using Moreau-Yosida Regularization
arXiv:2007.13322
Abstract
This research proposes to use the Moreau-Yosida envelope to stabilize the convergence behavior of bi-level Hyperparameter optimization solvers, and introduces the new algorithm called Moreau-Yosida regularized Hyperparameter Optimization (MY-HPO) algorithm. Theoretical analysis on the correctness of the MY-HPO solution and initial convergence analysis is also provided. Our empirical results show significant improvement in loss values for a fixed computation budget, compared to the state-of-art bi-level HPO solvers.
arXiv commentAutoML, Hyperparameter Optimization (HPO), Bi-Level Optimization, Alternating Direction Method of Multipliers (ADMM)
References in corpus (6)
- Practical Bayesian Optimization of Machine Learning Algorithms
- Bilevel Programming for Hyperparameter Optimization and Meta-Learning
- Stochastic Hyperparameter Optimization through Hypernetworks
- Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions
- Penalty Method for Inversion-Free Deep Bilevel Optimization
- Auptimizer -- an Extensible, Open-Source Framework for Hyperparameter Tuning