Bilevel Programming for Hyperparameter Optimization and Meta-Learning

Abstract

We introduce a framework based on bilevel programming that unifies gradient-based hyperparameter optimization and meta-learning. We show that an approximate version of the bilevel problem can be solved by taking into explicit account the optimization dynamics for the inner objective. Depending on the specific setting, the outer variables take either the meaning of hyperparameters in a supervised learning problem or parameters of a meta-learner. We provide sufficient conditions under which solutions of the approximate problem converge to those of the exact problem. We instantiate our approach for meta-learning in the case of deep learning where representation layers are treated as hyperparameters shared across a set of training episodes. In experiments, we confirm our theoretical findings, present encouraging results for few-shot learning and contrast the bilevel approach against classical approaches for learning-to-learn.

arXiv commentICML 2018; code for replicating experiments at https://github.com/prolearner/hyper-representation, main package (Far-HO) at https://github.com/lucfra/FAR-HO

References in corpus (12)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Practical Bayesian Optimization of Machine Learning Algorithms
• Prototypical Networks for Few-shot Learning
• Theoretical Models of Learning to Learn
• Matching Networks for One Shot Learning
• A Simple Neural Attentive Meta-Learner
• Gradient-based Hyperparameter Optimization through Reversible Learning
• Learning to Remember Rare Events
• Towards a Neural Statistician
• Hyperparameter optimization with approximate gradient
• Forward and Reverse Gradient-Based Hyperparameter Optimization
• Incremental Learning-to-Learn with Statistical Guarantees

Cited by in corpus (34)

• Generalized Inner Loop Meta-Learning
• Coresets via Bilevel Optimization for Continual Learning and Streaming
• Revisiting Adversarially Learned Injection Attacks Against Recommender Systems
• How to Retrain Recommender System? A Sequential Meta-Learning Method
• Online Hyperparameter Optimization for Class-Incremental Learning
• Variational Rectification Inference for Learning with Noisy Labels
• Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
• SGAS: Sequential Greedy Architecture Search
• How Important is the Train-Validation Split in Meta-Learning?
• Scalable Bayesian Meta-Learning through Generalized Implicit Gradients
• Network Architecture Search for Domain Adaptation
• A Gradient-based Bilevel Optimization Approach for Tuning Hyperparameters in Machine Learning
• Contrastive Learning Improves Model Robustness Under Label Noise
• Improved Bilevel Model: Fast and Optimal Algorithm with Theoretical Guarantee
• Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians
• Modeling and Optimization Trade-off in Meta-learning
• Safe Grid Search with Optimal Complexity
• Differentiable Meta-learning Model for Few-shot Semantic Segmentation
• Enhanced Bilevel Optimization via Bregman Distance
• Gradient-based algorithms for multi-objective bi-level optimization
• Efficient hyperparameter optimization by way of PAC-Bayes bound minimization
• Hyper-parameter Tuning under a Budget Constraint
• Efficient Gradient Approximation Method for Constrained Bilevel Optimization
• Bi-level Score Matching for Learning Energy-based Latent Variable Models
• How Out-of-Distribution Data Hurts Semi-Supervised Learning
• Bilevel Continual Learning
• Learning to Play Sequential Games versus Unknown Opponents
• Stabilizing Bi-Level Hyperparameter Optimization using Moreau-Yosida Regularization
• Do We Really Need Gold Samples for Sample Weighting Under Label Noise?
• Accelerating Gradient-based Meta Learner
• BOML: A Modularized Bilevel Optimization Library in Python for Meta Learning
• Towards Interpretable Multi-Task Learning Using Bilevel Programming
• Far-HO: A Bilevel Programming Package for Hyperparameter Optimization and Meta-Learning
• Optimizing Hyperparameters in CNNs using Bilevel Programming in Time Series Data