On the Theory of Transfer Learning: The Importance of Task Diversity

Abstract

We provide new statistical guarantees for transfer learning via representation learning--when transfer is achieved by learning a feature representation shared across different tasks. This enables learning on new tasks using far less data than is required to learn them in isolation. Formally, we consider $t+1$ tasks parameterized by functions of the form $f_j\circ h$ in a general function class $\mathcal{F}\circ \mathcal{H}$, where each $f_j$ is a task-specific function in $\mathcal{F}$ and $h$ is the shared representation in $\mathcal{H}$. Letting $C(\cdot )$ denote the complexity measure of the function class, we show that for diverse training tasks (1) the sample complexity needed to learn the shared representation across the first $t$ training tasks scales as $C(\mathcal{H})+tC(\mathcal{F})$, despite no explicit access to a signal from the feature representation and (2) with an accurate estimate of the representation, the sample complexity needed to learn a new task scales only with $C(\mathcal{F})$. Our results depend upon a new general notion of task diversity--applicable to models with general tasks, features, and losses--as well as a novel chain rule for Gaussian complexities. Finally, we exhibit the utility of our general framework in several models of importance in the literature.

arXiv commentNeurIPS 2020

References in corpus (11)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Support union recovery in high-dimensional multivariate regression
• The Benefit of Multitask Representation Learning
• Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
• Efficient Learning of Generalized Linear and Single Index Models with Isotonic Regression
• Few-Shot Learning via Learning the Representation, Provably
• Provable Meta-Learning of Linear Representations
• Learning-to-Learn Stochastic Gradient Descent with Biased Regularization
• Provable Guarantees for Gradient-Based Meta-Learning
• Excess risk bounds for multitask learning with trace norm regularization

Cited by in corpus (19)

• On the Opportunities and Risks of Foundation Models
• Exploiting Shared Representations for Personalized Federated Learning
• Few-Shot Learning via Learning the Representation, Provably
• Probing transfer learning with a model of synthetic correlated datasets
• Meta-Adaptive Nonlinear Control: Theory and Algorithms
• Meta-Learning with Graph Neural Networks: Methods and Applications
• How Important is the Train-Validation Split in Meta-Learning?
• Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?
• Weighted Training for Cross-Task Learning
• On the Power of Multitask Representation Learning in Linear MDP
• How Fine-Tuning Allows for Effective Meta-Learning
• A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?
• Adversarial Training Helps Transfer Learning via Better Representations
• On the Statistical Benefits of Curriculum Learning
• For Manifold Learning, Deep Neural Networks can be Locality Sensitive Hash Functions
• Provable Adaptation across Multiway Domains via Representation Learning
• Functional Regularization for Representation Learning: A Unified Theoretical Perspective
• Linear Speedup in Personalized Collaborative Learning
• Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks