Scalable Transfer Learning with Expert Models

Abstract

Transfer of pre-trained representations can improve sample efficiency and reduce computational requirements for new tasks. However, representations used for transfer are usually generic, and are not tailored to a particular distribution of downstream tasks. We explore the use of expert representations for transfer with a simple, yet effective, strategy. We train a diverse set of experts by exploiting existing label structures, and use cheap-to-compute performance proxies to select the relevant expert for each target task. This strategy scales the process of transferring to new tasks, since it does not revisit the pre-training data during transfer. Accordingly, it requires little extra compute per target task, and results in a speed-up of 2-3 orders of magnitude compared to competing approaches. Further, we provide an adapter-based architecture able to compress many experts into a single model. We evaluate our approach on two different data sources and demonstrate that it outperforms baselines on over 20 diverse vision tasks in both cases.

References in corpus (16)

• Distilling the Knowledge in a Neural Network
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• How transferable are features in deep neural networks?
• Deep Domain Confusion: Maximizing for Domain Invariance
• Fine-Grained Visual Classification of Aircraft
• Billion-scale semi-supervised learning for image classification
• Self-training with Noisy Student improves ImageNet classification
• A Closer Look at Few-shot Classification
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Parameter-Efficient Transfer Learning for NLP
• Learning Factored Representations in a Deep Mixture of Experts
• Taking Advantage of Sparsity in Multi-Task Learning
• Domain Adaptive Transfer Learning with Specialist Models
• Hyperbolic Discounting and Learning over Multiple Horizons
• Hypothesis Transfer Learning via Transformation Functions
• Theoretical Guarantees of Transfer Learning

Cited by in corpus (9)

• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Exploring the Limits of Large Scale Pre-training
• Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts
• Learning a Universal Template for Few-shot Dataset Generalization
• Deep Ensembles for Low-Data Transfer Learning
• ImageNet-21K Pretraining for the Masses
• Ranking Neural Checkpoints
• Representation Consolidation for Training Expert Students
• Sequential Random Network for Fine-grained Image Classification