A linearized framework and a new benchmark for model selection for fine-tuning

Abstract

Fine-tuning from a collection of models pre-trained on different domains (a "model zoo") is emerging as a technique to improve test accuracy in the low-data regime. However, model selection, i.e. how to pre-select the right model to fine-tune from a model zoo without performing any training, remains an open topic. We use a linearized framework to approximate fine-tuning, and introduce two new baselines for model selection -- Label-Gradient and Label-Feature Correlation. Since all model selection algorithms in the literature have been tested on different use-cases and never compared directly, we introduce a new comprehensive benchmark for model selection comprising of: i) A model zoo of single and multi-domain models, and ii) Many target tasks. Our benchmark highlights accuracy gain with model zoo compared to fine-tuning Imagenet models. We show our model selection baseline can select optimal models to fine-tune in few selections and has the highest ranking correlation to fine-tuning accuracy compared to existing algorithms.

14 pages

References in corpus (9)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Densely Connected Convolutional Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Rethinking Pre-training and Self-training
• LEEP: A New Measure to Evaluate Transferability of Learned Representations
• Rethinking the Hyperparameters for Fine-tuning
• Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network
• Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
• Enhancing Flood Impact Analysis using Interactive Retrieval of Social Media Images

Cited by in corpus (6)

• What can linearized neural networks actually say about generalization?
• Frustratingly Easy Transferability Estimation
• Newer is not always better: Rethinking transferability metrics, their peculiarities, stability and performance
• Representation Consolidation for Training Expert Students
• Unsupervised Model Drift Estimation with Batch Normalization Statistics for Dataset Shift Detection and Model Selection
• DIVA: Dataset Derivative of a Learning Task