Phase Transitions in Transfer Learning for High-Dimensional Perceptrons

Abstract

Transfer learning seeks to improve the generalization performance of a target task by exploiting the knowledge learned from a related source task. Central questions include deciding what information one should transfer and when transfer can be beneficial. The latter question is related to the so-called negative transfer phenomenon, where the transferred source information actually reduces the generalization performance of the target task. This happens when the two tasks are sufficiently dissimilar. In this paper, we present a theoretical analysis of transfer learning by studying a pair of related perceptron learning tasks. Despite the simplicity of our model, it reproduces several key phenomena observed in practice. Specifically, our asymptotic analysis reveals a phase transition from negative transfer to positive transfer as the similarity of the two tasks moves past a well-defined threshold.

References in corpus (9)

• A framework to characterize performance of LASSO algorithms
• An analytic theory of generalization dynamics and transfer learning in deep linear networks
• Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization
• Phase Retrieval via Polytope Optimization: Geometry, Phase Transitions, and New Algorithms
• A Precise Performance Analysis of Learning with Random Features
• Precise Error Analysis of Regularized M-estimators in High-dimensions
• The role of regularization in classification of high-dimensional noisy Gaussian mixture
• Solvable Model for Inheriting the Regularization through Knowledge Distillation
• Double Double Descent: On Generalization Errors in Transfer Learning between Linear Regression Tasks

Cited by in corpus (8)

• Probing transfer learning with a model of synthetic correlated datasets
• Gaussian Universality of Perceptrons with Random Labels
• On the Inherent Regularization Effects of Noise Injection During Training
• Eight challenges in developing theory of intelligence
• Statistical mechanics of continual learning: variational principle and mean-field potential
• A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning
• The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensions
• Optimal Protocols for Continual Learning via Statistical Physics and Control Theory