Ranking Neural Checkpoints
arXiv:2011.11200
Abstract
This paper is concerned with ranking many pre-trained deep neural networks (DNNs), called checkpoints, for the transfer learning to a downstream task. Thanks to the broad use of DNNs, we may easily collect hundreds of checkpoints from various sources. Which of them transfers the best to our downstream task of interest? Striving to answer this question thoroughly, we establish a neural checkpoint ranking benchmark (NeuCRaB) and study some intuitive ranking measures. These measures are generic, applying to the checkpoints of different output types without knowing how the checkpoints are pre-trained on which dataset. They also incur low computation cost, making them practically meaningful. Our results suggest that the linear separability of the features extracted by the checkpoints is a strong indicator of transferability. We also arrive at a new ranking measure, NLEEP, which gives rise to the best performance in the experiments.
arXiv commentAccepted to CVPR 2021
References in corpus (23)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Decoupled Weight Decay Regularization
- A Simple Framework for Contrastive Learning of Visual Representations
- DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Bootstrap your own latent: A new approach to self-supervised Learning
- On Calibration of Modern Neural Networks
- Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
- Semi-Supervised Learning with Ladder Networks
- Domain Adaptation: Learning Bounds and Algorithms
- Rethinking Pre-training and Self-training
- Learning with Pseudo-Ensembles
- Robust Large Margin Deep Neural Networks
- Wasserstein Auto-Encoders
- A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
- Predicting the Generalization Gap in Deep Networks with Margin Distributions
- Stronger generalization bounds for deep nets via a compression approach
- LEEP: A New Measure to Evaluate Transferability of Learned Representations
- Geometric Dataset Distances via Optimal Transport
- Scalable Transfer Learning with Expert Models
- Deep Model Transferability from Attribution Maps
- Towards Task and Architecture-Independent Generalization Gap Predictors