Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis

Abstract

Bilevel optimization has become a powerful framework in various machine learning applications including meta-learning, hyperparameter optimization, and network architecture search. There are generally two classes of bilevel optimization formulations for machine learning: 1) problem-based bilevel optimization, whose inner-level problem is formulated as finding a minimizer of a given loss function; and 2) algorithm-based bilevel optimization, whose inner-level solution is an output of a fixed algorithm. For the first class, two popular types of gradient-based algorithms have been proposed for hypergradient estimation via approximate implicit differentiation (AID) and iterative differentiation (ITD). Algorithms for the second class include the popular model-agnostic meta-learning (MAML) and almost no inner loop (ANIL). However, the convergence rate and fundamental limitations of bilevel optimization algorithms have not been well explored. This thesis provides a comprehensive convergence rate analysis for bilevel algorithms in the aforementioned two classes. We further propose principled algorithm designs for bilevel optimization with higher efficiency and scalability. For the problem-based formulation, we provide a convergence rate analysis for AID- and ITD-based bilevel algorithms. We then develop acceleration bilevel algorithms, for which we provide shaper convergence analysis with relaxed assumptions. We also provide the first lower bounds for bilevel optimization, and establish the optimality by providing matching upper bounds under certain conditions. We finally propose new stochastic bilevel optimization algorithms with lower complexity and higher efficiency in practice. For the algorithm-based formulation, we develop a theoretical convergence for general multi-step MAML and ANIL, and characterize the impact of parameter selections and loss geometries on the their complexities.

arXiv commentPh.D. dissertation, 253 pages, 7 tables and 8 figures. All comments are welcome

References in corpus (35)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• DARTS: Differentiable Architecture Search
• Theoretical Models of Learning to Learn
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• On First-Order Meta-Learning Algorithms
• Hyper-Parameter Optimization: A Review of Algorithms and Applications
• Adversarial Attacks on Graph Neural Networks via Meta Learning
• On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimization
• Deep Meta-Learning: Learning to Learn in the Concept Space
• Online Meta-Learning
• Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimization
• BOIL: Towards Representation Change for Few-shot Learning
• Approximation Methods for Bilevel Programming
• Few-Shot Learning via Learning the Representation, Provably
• A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Critic
• Provable Meta-Learning of Linear Representations
• Learning-to-Learn Stochastic Gradient Descent with Biased Regularization
• Near-Optimal Algorithms for Minimax Optimization
• On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithms
• Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions
• A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentum
• Penalty Method for Inversion-Free Deep Bilevel Optimization
• Incremental Learning-to-Learn with Statistical Guarantees
• Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization
• Lower Bounds and Accelerated Algorithms for Bilevel Optimization
• When Will Gradient Methods Converge to Max-margin Classifier under ReLU Models?
• A Gradient-based Bilevel Optimization Approach for Tuning Hyperparameters in Machine Learning
• Improved Bilevel Model: Fast and Optimal Algorithm with Theoretical Guarantee
• Provably Faster Algorithms for Bilevel Optimization
• Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians
• Boosting One-Point Derivative-Free Online Optimization via Residual Feedback
• On the Global Optimality of Model-Agnostic Meta-Learning
• Learning Latent Features with Pairwise Penalties in Low-Rank Matrix Completion
• Multi-step Estimation for Gradient-based Meta-learning