Minimax Lower Bounds for Transfer Learning with Linear and One-hidden Layer Neural Networks

Abstract

Transfer learning has emerged as a powerful technique for improving the performance of machine learning models on new domains where labeled training data may be scarce. In this approach a model trained for a source task, where plenty of labeled training data is available, is used as a starting point for training a model on a related target task with only few labeled training data. Despite recent empirical success of transfer learning approaches, the benefits and fundamental limits of transfer learning are poorly understood. In this paper we develop a statistical minimax framework to characterize the fundamental limits of transfer learning in the context of regression with linear and one-hidden layer neural network models. Specifically, we derive a lower-bound for the target generalization error achievable by any algorithm as a function of the number of labeled source and target data as well as appropriate notions of similarity between the source and target tasks. Our lower bound provides new insights into the benefits and limitations of transfer learning. We further corroborate our theoretical finding with various experiments.

References in corpus (9)

• Unsupervised Domain Adaptation with Residual Transfer Networks
• Domain Adaptation: Learning Bounds and Algorithms
• Wasserstein Distance Guided Representation Learning for Domain Adaptation
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• On Learning Invariant Representation for Domain Adaptation
• Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation
• Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment
• Regularized Learning for Domain Adaptation under Label Shifts
• On the Value of Target Data in Transfer Learning

Cited by in corpus (5)

• Explaining the physics of transfer learning a data-driven subgrid-scale closure to a different turbulent flow
• Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?
• Transfer-Learning Across Datasets with Different Input Dimensions: An Algorithm and Analysis for the Linear Regression Case
• How Does the Task Landscape Affect MAML Performance?
• On the Statistical Benefits of Curriculum Learning