Probing transfer learning with a model of synthetic correlated datasets

Abstract

Transfer learning can significantly improve the sample efficiency of neural networks, by exploiting the relatedness between a data-scarce target task and a data-abundant source task. Despite years of successful applications, transfer learning practice often relies on ad-hoc solutions, while theoretical understanding of these procedures is still limited. In the present work, we re-think a solvable model of synthetic data as a framework for modeling correlation between data-sets. This setup allows for an analytic characterization of the generalization performance obtained when transferring the learned feature map from the source to the target task. Focusing on the problem of training two-layer networks in a binary classification setting, we show that our model can capture a range of salient features of transfer learning with real data. Moreover, by exploiting parametric control over the correlation between the two data-sets, we systematically investigate under which conditions the transfer of features is beneficial for generalization.

References in corpus (36)

• Scikit-learn: Machine Learning in Python
• Auto-Encoding Variational Bayes
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• How transferable are features in deep neural networks?
• Language Models are Few-Shot Learners
• On the Opportunities and Risks of Foundation Models
• GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification
• Reconciling modern machine learning practice and the bias-variance trade-off
• Theoretical Models of Learning to Learn
• Generalizing from a Few Examples: A Survey on Few-Shot Learning
• Transfusion: Understanding Transfer Learning for Medical Imaging
• A Model of Inductive Bias Learning
• Estimating the intrinsic dimension of datasets by a minimal neighborhood information
• The generalization error of random features regression: Precise asymptotics and double descent curve
• What is being transferred in transfer learning?
• Modelling the influence of data structure on learning in neural networks: the hidden manifold model
• Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
• Generalisation error in learning with random features and the hidden manifold model
• A jamming transition from under- to over-parametrization affects loss landscape and generalization
• On the Theory of Transfer Learning: The Importance of Task Diversity
• Geometric Dataset Distances via Optimal Transport
• Transfer Adaptation Learning: A Decade Survey
• Few-Shot Learning via Learning the Representation, Provably
• Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime
• Disentangling feature and lazy training in deep neural networks
• An analytic theory of generalization dynamics and transfer learning in deep linear networks
• The Gaussian equivalence of generative models for learning with shallow neural networks
• Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm
• Towards Automated Melanoma Screening: Exploring Transfer Learning Schemes
• Statistical learning theory of structured data
• Universality Laws for High-Dimensional Learning with Random Features
• DeepRadiologyNet: Radiologist Level Pathology Detection in CT Head Images
• Phase Transitions in Transfer Learning for High-Dimensional Perceptrons
• Similarity of Classification Tasks
• An Information-Geometric Distance on the Space of Tasks
• Double Double Descent: On Generalization Errors in Transfer Learning between Linear Regression Tasks

Cited by in corpus (6)

• Gaussian Universality of Perceptrons with Random Labels
• A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning
• Daydreaming Hopfield Networks and their surprising effectiveness on correlated data
• The impact of memory on learning sequence-to-sequence tasks
• Optimal Protocols for Continual Learning via Statistical Physics and Control Theory
• The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensions