Deep Ensembles for Low-Data Transfer Learning
arXiv:2010.06866
Abstract
In the low-data regime, it is difficult to train good supervised models from scratch. Instead practitioners turn to pre-trained models, leveraging transfer learning. Ensembling is an empirically and theoretically appealing way to construct powerful predictive models, but the predominant approach of training multiple deep networks with different random initialisations collides with the need for transfer via pre-trained weights. In this work, we study different ways of creating ensembles from pre-trained models. We show that the nature of pre-training itself is a performant source of diversity, and propose a practical algorithm that efficiently identifies a subset of pre-trained models for any downstream dataset. The approach is simple: Use nearest-neighbour accuracy to rank pre-trained models, fine-tune the best ones with a small hyperparameter sweep, and greedily construct an ensemble to minimise validation cross-entropy. When evaluated together with strong baselines on 19 different downstream tasks (the Visual Task Adaptation Benchmark), this achieves state-of-the-art performance at a much lower inference budget, even when selecting from over 2,000 pre-trained models. We also assess our ensembles on ImageNet variants and show improved robustness to distribution shift.
References in corpus (15)
- AutoML: A Survey of the State-of-the-Art
- Do ImageNet Classifiers Generalize to ImageNet?
- Learning with Pseudo-Ensembles
- Deep Ensembles: A Loss Landscape Perspective
- Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
- Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
- What is being transferred in transfer learning?
- Natural Adversarial Examples
- A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
- BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning
- Domain Adaptive Transfer Learning with Specialist Models
- Superposition of many models into one
- Hyperparameter Ensembles for Robustness and Uncertainty Quantification
- Scalable Transfer Learning with Expert Models
- Diverse Ensembles Improve Calibration