Domain Adaptive Transfer Learning with Specialist Models

Abstract

Transfer learning is a widely used method to build high performing computer vision models. In this paper, we study the efficacy of transfer learning by examining how the choice of data impacts performance. We find that more pre-training data does not always help, and transfer performance depends on a judicious choice of pre-training data. These findings are important given the continued increase in dataset sizes. We further propose domain adaptive transfer learning, a simple and effective pre-training method using importance weights computed based on the target dataset. Our method to compute importance weights follow from ideas in domain adaptation, and we show a novel application to transfer learning. Our methods achieve state-of-the-art results on multiple fine-grained classification datasets and are well-suited for use in practice.

References in corpus (6)

• YFCC100M: The New Data in Multimedia Research
• CNN Features off-the-shelf: an Astounding Baseline for Recognition
• Rethinking the Inception Architecture for Computer Vision
• Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
• Speed/accuracy trade-offs for modern convolutional object detectors
• Analyzing the Performance of Multilayer Neural Networks for Object Recognition

Cited by in corpus (28)

• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Transfusion: Understanding Transfer Learning for Medical Imaging
• Self-training with Noisy Student improves ImageNet classification
• What is being transferred in transfer learning?
• SpinalNet: Deep Neural Network with Gradual Input
• Sharpness-Aware Minimization for Efficiently Improving Generalization
• Rethinking the Hyperparameters for Fine-tuning
• A Survey of Deep Learning for Scientific Discovery
• Data Valuation using Reinforcement Learning
• Leveraging Siamese Networks for One-Shot Intrusion Detection Model
• Exploring the Limits of Large Scale Pre-training
• In-domain representation learning for remote sensing
• Scalable Transfer Learning with Expert Models
• Self-training for Few-shot Transfer Across Extreme Task Differences
• When Does Self-supervision Improve Few-shot Learning?
• Deep Ensembles for Low-Data Transfer Learning
• Adaptive Consistency Regularization for Semi-Supervised Transfer Learning
• Measuring Dataset Granularity
• Auxiliary Task Update Decomposition: The Good, The Bad and The Neutral
• Non-binary deep transfer learning for image classification
• ImageNet-21K Pretraining for the Masses
• Optimizing Data Usage via Differentiable Rewards
• Classification of Melanocytic Nevus Images using BigTransfer (BiT)
• Sequential Random Network for Fine-grained Image Classification
• Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation
• Chair Segments: A Compact Benchmark for the Study of Object Segmentation
• Learning to Transfer Learn: Reinforcement Learning-Based Selection for Adaptive Transfer Learning
• Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement