Learn Faster and Forget Slower via Fast and Stable Task Adaptation

Abstract

Training Deep Neural Networks (DNNs) is still highly time-consuming and compute-intensive. It has been shown that adapting a pretrained model may significantly accelerate this process. With a focus on classification, we show that current fine-tuning techniques make the pretrained models catastrophically forget the transferred knowledge even before anything about the new task is learned. Such rapid knowledge loss undermines the merits of transfer learning and may result in a much slower convergence rate compared to when the maximum amount of knowledge is exploited. We investigate the source of this problem from different perspectives and to alleviate it, introduce Fast And Stable Task-adaptation (FAST), an easy to apply fine-tuning algorithm. The paper provides a novel geometric perspective on how the loss landscape of source and target tasks are linked in different transfer learning strategies. We empirically show that compared to prevailing fine-tuning practices, FAST learns the target task faster and forgets the source task slower.

arXiv comment52 pages, 15 figures, 1 table

References in corpus (37)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• ADADELTA: An Adaptive Learning Rate Method
• Prototypical Networks for Few-shot Learning
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Densely Connected Convolutional Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• On the Convergence of Adam and Beyond
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• On the Variance of the Adaptive Learning Rate and Beyond
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• Three scenarios for continual learning
• On First-Order Meta-Learning Algorithms
• Improving Generalization Performance by Switching from Adam to SGD
• Training Tips for the Transformer Model
• Self-training with Noisy Student improves ImageNet classification
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• All you need is a good init
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Essentially No Barriers in Neural Network Energy Landscape
• Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
• Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• Analyzing the Performance of Multilayer Neural Networks for Object Recognition
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• Continuous Domain Adaptation with Variational Domain-Agnostic Feature Replay
• Geometry of energy landscapes and the optimizability of deep neural networks
• The Benefits of Over-parameterization at Initialization in Deep ReLU Networks
• Transferring Inductive Biases through Knowledge Distillation
• Loss Patterns of Neural Networks
• Interpreting and Evaluating Neural Network Robustness
• A Deep Neural Network's Loss Surface Contains Every Low-dimensional Pattern
• Efficient Neural Task Adaptation by Maximum Entropy Initialization