Learning-to-Learn Stochastic Gradient Descent with Biased Regularization

Abstract

We study the problem of learning-to-learn: inferring a learning algorithm that works well on tasks sampled from an unknown distribution. As class of algorithms we consider Stochastic Gradient Descent on the true risk regularized by the square euclidean distance to a bias vector. We present an average excess risk bound for such a learning algorithm. This result quantifies the potential benefit of using a bias vector with respect to the unbiased case. We then address the problem of estimating the bias from a sequence of tasks. We propose a meta-algorithm which incrementally updates the bias, as new tasks are observed. The low space and time complexity of this approach makes it appealing in practice. We provide guarantees on the learning ability of the meta-algorithm. A key feature of our results is that, when the number of tasks grows and their variance is relatively small, our learning-to-learn approach has a significant advantage over learning each task in isolation by Stochastic Gradient Descent without a bias term. We report on numerical experiments which demonstrate the effectiveness of our approach.

37 pages, 8 figures

Cited by in corpus (29)

• On the Theory of Transfer Learning: The Importance of Task Diversity
• Few-Shot Learning via Learning the Representation, Provably
• Provable Meta-Learning of Linear Representations
• Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learning
• On the Iteration Complexity of Hypergradient Computation
• Differentially Private Meta-Learning
• Meta-Adaptive Nonlinear Control: Theory and Algorithms
• When MAML Can Adapt Fast and How to Assist When It Cannot
• Generalization Bounds For Meta-Learning: An Information-Theoretic Analysis
• Meta-Learning with Graph Neural Networks: Methods and Applications
• Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?
• Modular Meta-Learning with Shrinkage
• A Sample Complexity Separation between Non-Convex and Convex Meta-Learning
• Distance-Based Regularisation of Deep Networks for Fine-Tuning
• Convergence Properties of Stochastic Hypergradients
• On the Power of Multitask Representation Learning in Linear MDP
• Provable Representation Learning for Imitation Learning via Bi-level Optimization
• Impact of Representation Learning in Linear Bandits
• Conditional Meta-Learning of Linear Representations
• Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis
• The Advantage of Conditional Meta-Learning for Biased Regularization and Fine-Tuning
• Theoretical bounds on estimation error for meta-learning
• Meta-Learning with Network Pruning
• A Distribution-Dependent Analysis of Meta-Learning
• Learning-to-learn non-convex piecewise-Lipschitz functions
• Accelerating Distributed Online Meta-Learning via Multi-Agent Collaboration under Limited Communication
• Dynamic Regret Analysis for Online Meta-Learning
• Covariate Distribution Aware Meta-learning
• On Data Efficiency of Meta-learning