Provable Meta-Learning of Linear Representations

Abstract

Meta-learning, or learning-to-learn, seeks to design algorithms that can utilize previous experience to rapidly learn new skills or adapt to new environments. Representation learning -- a key tool for performing meta-learning -- learns a data representation that can transfer knowledge across multiple tasks, which is essential in regimes where data is scarce. Despite a recent surge of interest in the practice of meta-learning, the theoretical underpinnings of meta-learning algorithms are lacking, especially in the context of learning transferable representations. In this paper, we focus on the problem of multi-task linear regression -- in which multiple linear regression models share a common, low-dimensional linear representation. Here, we provide provably fast, sample-efficient algorithms to address the dual challenges of (1) learning a common set of features from multiple, related tasks, and (2) transferring this knowledge to new, unseen tasks. Both are central to the general problem of meta-learning. Finally, we complement these results by providing information-theoretic lower bounds on the sample complexity of learning these linear features.

Lower bound slightly improved to include task diversity parameter

References in corpus (4)

• Multi-Task Deep Neural Networks for Natural Language Understanding
• Few-Shot Learning via Learning the Representation, Provably
• Learning-to-Learn Stochastic Gradient Descent with Biased Regularization
• Provable Guarantees for Gradient-Based Meta-Learning

Cited by in corpus (29)

• Exploiting Shared Representations for Personalized Federated Learning
• On the Theory of Transfer Learning: The Importance of Task Diversity
• Few-Shot Learning via Learning the Representation, Provably
• What Makes Multi-modal Learning Better than Single (Provably)
• Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learning
• Meta-Adaptive Nonlinear Control: Theory and Algorithms
• Meta-Learning with Graph Neural Networks: Methods and Applications
• Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?
• How Important is the Train-Validation Split in Meta-Learning?
• Meta-Learning with Fewer Tasks through Task Interpolation
• Weighted Training for Cross-Task Learning
• Robust Meta-learning for Mixed Linear Regression with Small Batches
• Near-optimal Representation Learning for Linear Bandits and Linear RL
• On the Power of Multitask Representation Learning in Linear MDP
• Meta-Learning Bandit Policies by Gradient Ascent
• How Fine-Tuning Allows for Effective Meta-Learning
• Adversarial Training Helps Transfer Learning via Better Representations
• Impact of Representation Learning in Linear Bandits
• Meta-Learning with Neural Tangent Kernels
• The Advantage of Conditional Meta-Learning for Biased Regularization and Fine-Tuning
• Conditional Meta-Learning of Linear Representations
• Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis
• Optimal Multitask Linear Regression and Contextual Bandits under Sparse Heterogeneity
• How Does the Task Landscape Affect MAML Performance?
• Sample Efficient Linear Meta-Learning by Alternating Minimization
• Meta-learning Transferable Representations with a Single Target Domain
• Learning Mixtures of Low-Rank Models
• Provable Lifelong Learning of Representations
• Online Parameter-Free Learning of Multiple Low Variance Tasks