Rethinking the Hyperparameters for Fine-tuning

Abstract

Fine-tuning from pre-trained ImageNet models has become the de-facto standard for various computer vision tasks. Current practices for fine-tuning typically involve selecting an ad-hoc choice of hyperparameters and keeping them fixed to values normally used for training from scratch. This paper re-examines several common practices of setting hyperparameters for fine-tuning. Our findings are based on extensive empirical evaluation for fine-tuning on various transfer learning benchmarks. (1) While prior works have thoroughly investigated learning rate and batch size, momentum for fine-tuning is a relatively unexplored parameter. We find that the value of momentum also affects fine-tuning performance and connect it with previous theoretical findings. (2) Optimal hyperparameters for fine-tuning, in particular, the effective learning rate, are not only dataset dependent but also sensitive to the similarity between the source domain and target domain. This is in contrast to hyperparameters for training from scratch. (3) Reference-based regularization that keeps models close to the initial model does not necessarily apply for "dissimilar" datasets. Our findings challenge common practices of fine-tuning and encourages deep learning practitioners to rethink the hyperparameters for fine-tuning.

Published as a conference paper at ICLR 2020

References in corpus (15)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Decoupled Weight Decay Regularization
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• How transferable are features in deep neural networks?
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Fine-Grained Visual Classification of Aircraft
• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• AutoAugment: Learning Augmentation Policies from Data
• What makes ImageNet good for transfer learning?
• Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
• Explicit Inductive Bias for Transfer Learning with Convolutional Networks
• Domain Adaptive Transfer Learning with Specialist Models
• Three Mechanisms of Weight Decay Regularization

Cited by in corpus (19)

• A review of federated learning in renewable energy applications: Potential, challenges, and future directions
• AdvParams: An Active DNN Intellectual Property Protection Technique via Adversarial Perturbation Based Parameter Encryption
• LogME: Practical Assessment of Pre-trained Models for Transfer Learning
• Pretraining Representations for Data-Efficient Reinforcement Learning
• AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks
• SCSS-Net: Solar Corona Structures Segmentation by Deep Learning
• FREE: Feature Refinement for Generalized Zero-Shot Learning
• Newer is not always better: Rethinking transferability metrics, their peculiarities, stability and performance
• Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubs
• A linearized framework and a new benchmark for model selection for fine-tuning
• An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization
• Non-binary deep transfer learning for image classification
• Low-Fidelity End-to-End Video Encoder Pre-training for Temporal Action Localization
• Sample-based Regularization: A Transfer Learning Strategy Toward Better Generalization
• Robust and Explainable Autoencoders for Unsupervised Time Series Outlier Detection---Extended Version
• Modular Adaptation for Cross-Domain Few-Shot Learning
• Renofeation: A Simple Transfer Learning Method for Improved Adversarial Robustness
• Adapting Machine Learning Diagnostic Models to New Populations Using a Small Amount of Data: Results from Clinical Neuroscience
• DIVA: Dataset Derivative of a Learning Task