Meta-learning Transferable Representations with a Single Target Domain

Abstract

Recent works found that fine-tuning and joint training---two popular approaches for transfer learning---do not always improve accuracy on downstream tasks. First, we aim to understand more about when and why fine-tuning and joint training can be suboptimal or even harmful for transfer learning. We design semi-synthetic datasets where the source task can be solved by either source-specific features or transferable features. We observe that (1) pre-training may not have incentive to learn transferable features and (2) joint training may simultaneously learn source-specific features and overfit to the target. Second, to improve over fine-tuning and joint training, we propose Meta Representation Learning (MeRLin) to learn transferable features. MeRLin meta-learns representations by ensuring that a head fit on top of the representations with target training data also performs well on target validation data. We also prove that MeRLin recovers the target ground-truth model with a quadratic neural net parameterization and a source distribution that contains both transferable and source-specific features. On the same distribution, pre-training and joint training provably fail to learn transferable features. MeRLin empirically outperforms previous state-of-the-art transfer learning algorithms on various real-world vision and NLP transfer learning benchmarks.

References in corpus (16)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Neural Architecture Search with Reinforcement Learning
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• How transferable are features in deep neural networks?
• Learning Transferable Features with Deep Adaptation Networks
• Theoretical Models of Learning to Learn
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• What makes ImageNet good for transfer learning?
• Deep Equilibrium Models
• Meta-Learning with Implicit Gradients
• What is being transferred in transfer learning?
• Rethinking ImageNet Pre-training
• Provable Meta-Learning of Linear Representations
• Understanding and Improving Information Transfer in Multi-Task Learning
• Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks
• Towards Understanding the Transferability of Deep Representations

Cited by in corpus (1)

• Weighted Training for Cross-Task Learning