What is being transferred in transfer learning?

Abstract

One desired capability for machines is the ability to transfer their knowledge of one domain to another where data is (usually) scarce. Despite ample adaptation of transfer learning in various deep learning applications, we yet do not understand what enables a successful transfer and which part of the network is responsible for that. In this paper, we provide new tools and analyses to address these fundamental questions. Through a series of analyses on transferring to block-shuffled images, we separate the effect of feature reuse from learning low-level statistics of data and show that some benefit of transfer learning comes from the latter. We present that when training from pre-trained weights, the model stays in the same basin in the loss landscape and different instances of such model are similar in feature space and close in parameter space.

arXiv commentEqual contribution, authors ordered randomly

References in corpus (16)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• How transferable are features in deep neural networks?
• Language Models are Few-Shot Learners
• CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Similarity of Neural Network Representations Revisited
• What makes ImageNet good for transfer learning?
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Essentially No Barriers in Neural Network Energy Landscape
• Fantastic Generalization Measures and Where to Find Them
• The Singular Values of Convolutional Layers
• Fixup Initialization: Residual Learning Without Normalization
• Are All Layers Created Equal?
• Rethinking ImageNet Pre-training
• Domain Adaptive Transfer Learning with Specialist Models

Cited by in corpus (19)

• A Review of Deep Transfer Learning and Recent Advancements
• EXPANSE: A Deep Continual / Progressive Learning System for Deep Transfer Learning
• Embrace Limited and Imperfect Training Datasets: Opportunities and Challenges in Plant Disease Recognition Using Deep Learning
• Hybrid quantum physics-informed neural networks for simulating computational fluid dynamics in complex shapes
• Probing transfer learning with a model of synthetic correlated datasets
• An analysis of the transfer learning of convolutional neural networks for artistic images
• Transferring Inductive Biases through Knowledge Distillation
• FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning
• Deep Ensembles for Low-Data Transfer Learning
• Closer Look at the Uncertainty Estimation in Semantic Segmentation under Distributional Shift
• Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic Forgetting in Automatic Speech Recognition
• No One Representation to Rule Them All: Overlapping Features of Training Methods
• CoReS: Compatible Representations via Stationarity
• Parameter Prediction for Unseen Deep Architectures
• Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models
• Meta-learning Transferable Representations with a Single Target Domain
• Chair Segments: A Compact Benchmark for the Study of Object Segmentation
• Weighted Ensemble Models Are Strong Continual Learners
• Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Data