LogME: Practical Assessment of Pre-trained Models for Transfer Learning

Abstract

This paper studies task adaptive pre-trained model selection, an underexplored problem of assessing pre-trained models for the target task and select best ones from the model zoo \emph{without fine-tuning}. A few pilot works addressed the problem in transferring supervised pre-trained models to classification tasks, but they cannot handle emerging unsupervised pre-trained models or regression tasks. In pursuit of a practical assessment method, we propose to estimate the maximum value of label evidence given features extracted by pre-trained models. Unlike the maximum likelihood, the maximum evidence is \emph{immune to over-fitting}, while its expensive computation can be dramatically reduced by our carefully designed algorithm. The Logarithm of Maximum Evidence (LogME) can be used to assess pre-trained models for transfer learning: a pre-trained model with a high LogME value is likely to have good transfer performance. LogME is \emph{fast, accurate, and general}, characterizing itself as the first practical method for assessing pre-trained models. Compared with brute-force fine-tuning, LogME brings at most $3000\times$ speedup in wall-clock time and requires only $1\%$ memory footprint. It outperforms prior methods by a large margin in their setting and is applicable to new settings. It is general enough for diverse pre-trained models (supervised pre-trained and unsupervised pre-trained), downstream tasks (classification and regression), and modalities (vision and language). Code is available at this repository: \href{https://github.com/thuml/LogME}{https://github.com/thuml/LogME}.

arXiv comment13 pages (ICML 2021 camera ready version)

References in corpus (18)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• TensorFlow: A system for large-scale machine learning
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• How transferable are features in deep neural networks?
• Learning Transferable Features with Deep Adaptation Networks
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Improved Baselines with Momentum Contrastive Learning
• Fine-Grained Visual Classification of Aircraft
• What Makes for Good Views for Contrastive Learning?
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Strategies for Pre-training Graph Neural Networks
• Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Bayesian Evidence and Model Selection
• LEEP: A New Measure to Evaluate Transferability of Learned Representations
• Rethinking the Hyperparameters for Fine-tuning
• Semi-analytical approximations to statistical moments of sigmoid and softmax mappings of normal variables

Cited by in corpus (7)

• Training-free Lexical Backdoor Attacks on Language Models
• Frustratingly Easy Transferability Estimation
• Newer is not always better: Rethinking transferability metrics, their peculiarities, stability and performance
• Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubs
• How to Estimate Model Transferability of Pre-Trained Speech Models?
• ModelPS: An Interactive and Collaborative Platform for Editing Pre-trained Models at Scale
• On-target Adaptation