Improved Bilevel Model: Fast and Optimal Algorithm with Theoretical Guarantee
arXiv:2009.00690
Abstract
Due to the hierarchical structure of many machine learning problems, bilevel programming is becoming more and more important recently, however, the complicated correlation between the inner and outer problem makes it extremely challenging to solve. Although several intuitive algorithms based on the automatic differentiation have been proposed and obtained success in some applications, not much attention has been paid to finding the optimal formulation of the bilevel model. Whether there exists a better formulation is still an open problem. In this paper, we propose an improved bilevel model which converges faster and better compared to the current formulation. We provide theoretical guarantee and evaluation results over two tasks: Data Hyper-Cleaning and Hyper Representation Learning. The empirical results show that our model outperforms the current bilevel model with a great margin. \emph{This is a concurrent work with \citet{liu2020generic} and we submitted to ICML 2020. Now we put it on the arxiv for record.}
submitted to ICML2020
References in corpus (12)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Hyperparameter optimization with approximate gradient
- Bilevel Programming for Hyperparameter Optimization and Meta-Learning
- Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
- Stochastic Hyperparameter Optimization through Hypernetworks
- Online Learning Rate Adaptation with Hypergradient Descent
- Approximation Methods for Bilevel Programming
- Truncated Back-propagation for Bilevel Optimization
Cited by in corpus (4)
- A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Critic
- Communication-Efficient Robust Federated Learning with Noisy Labels
- Lower Bounds and Accelerated Algorithms for Bilevel Optimization
- Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis