What makes ImageNet good for transfer learning?

Abstract

The tremendous success of ImageNet-trained deep features on a wide range of transfer tasks begs the question: what are the properties of the ImageNet dataset that are critical for learning good, general-purpose features? This work provides an empirical investigation of various facets of this question: Is more pre-training data always better? How does feature quality depend on the number of training examples per class? Does adding more object classes improve performance? For the same data budget, how should the data be split into classes? Is fine-grained recognition necessary for learning good features? Given the same number of training classes, is it better to have coarse classes or fine-grained classes? Which is better: more classes or more examples per class? To answer these and related questions, we pre-trained CNN features on various subsets of the ImageNet dataset and evaluated transfer performance on PASCAL detection, PASCAL action classification, and SUN scene classification tasks. Our overall findings suggest that most changes in the choice of pre-training data long thought to be critical do not significantly affect transfer performance.? Given the same number of training classes, is it better to have coarse classes or fine-grained classes? Which is better: more classes or more examples per class?

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Residual Learning for Image Recognition
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Object Detectors Emerge in Deep Scene CNNs
• Understanding Deep Convolutional Networks
• CNN Features off-the-shelf: an Astounding Baseline for Recognition
• Unsupervised Visual Representation Learning by Context Prediction
• Unsupervised Learning of Visual Representations using Videos
• Learning to See by Moving
• Learning Representations for Automatic Colorization
• Instance-aware Semantic Segmentation via Multi-task Network Cascades
• Analyzing the Performance of Multilayer Neural Networks for Object Recognition
• Understanding Intra-Class Knowledge Inside CNN
• Unsupervised learning of object semantic parts from internal states of CNNs by population encoding
• Understanding deep features with computer-generated imagery
• Unsupervised Feature Learning from Temporal Data
• Digging Deep into the layers of CNNs: In Search of How CNNs Achieve View Invariance

Cited by in corpus (111)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Covid-19: Automatic detection from X-Ray images utilizing Transfer Learning with Convolutional Neural Networks
• Transfusion: Understanding Transfer Learning for Medical Imaging
• On the Compactness, Efficiency, and Representation of 3D Convolutional Networks: Brain Parcellation as a Pretext Task
• Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
• The EuroCity Persons Dataset: A Novel Benchmark for Object Detection
• Explainable COVID-19 Detection Using Chest CT Scans and Deep Learning
• What is being transferred in transfer learning?
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Parameter-Efficient Transfer Learning for NLP
• The Devil is in the Tails: Fine-grained Classification in the Wild
• Interventional Few-Shot Learning
• Do Adversarially Robust ImageNet Models Transfer Better?
• A critical analysis of self-supervision, or what we can learn from a single image
• DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networks
• Reading Wikipedia to Answer Open-Domain Questions
• What makes instance discrimination good for transfer learning?
• Would Mega-scale Datasets Further Enhance Spatiotemporal 3D CNNs?
• Overlearning Reveals Sensitive Attributes
• Scattering Networks for Hybrid Representation Learning
• Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation
• Rethinking the Hyperparameters for Fine-tuning
• Damage detection using in-domain and cross-domain transfer learning
• Analysis and Optimization of Convolutional Neural Network Architectures
• A Survey of Deep Learning for Scientific Discovery
• Overcoming Small Minirhizotron Datasets Using Transfer Learning
• What shapes feature representations? Exploring datasets, architectures, and training
• Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching
• Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning
• CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning
• An Analysis of Pre-Training on Object Detection
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• A Study of the Generalizability of Self-Supervised Representations
• A Survey on Machine Learning Techniques for Auto Labeling of Video, Audio, and Text Data
• Disentangling Image Distortions in Deep Feature Space
• CEREALS - Cost-Effective REgion-based Active Learning for Semantic Segmentation
• Improving Multi-Modal Learning with Uni-Modal Teachers
• Supervised Transfer Learning at Scale for Medical Imaging
• Self-supervised Video Representation Learning by Uncovering Spatio-temporal Statistics
• Locality and compositionality in zero-shot learning
• Dataset Distillation with Infinitely Wide Convolutional Networks
• Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation
• BREEDS: Benchmarks for Subpopulation Shift
• Scaling the Scattering Transform: Deep Hybrid Networks
• Few-Shot Image Recognition by Predicting Parameters from Activations
• The Next Big Thing(s) in Unsupervised Machine Learning: Five Lessons from Infant Learning
• CCBlock: An Effective Use of Deep Learning for Automatic Diagnosis of COVID-19 Using X-Ray Images
• Situational Fusion of Visual Representation for Visual Navigation
• Towards Understanding the Transferability of Deep Representations
• Improving Generalization via Scalable Neighborhood Component Analysis
• Self-Paced Contrastive Learning for Semi-supervised Medical Image Segmentation with Meta-labels
• Zero-Shot Learning from scratch (ZFS): leveraging local compositional representations
• Adaptive Consistency Regularization for Semi-Supervised Transfer Learning
• To Transfer or Not to Transfer: Misclassification Attacks Against Transfer Learned Text Classifiers
• Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark
• Now You See It, Now You Dont: Adversarial Vulnerabilities in Computational Pathology
• Unsupervised Joint Mining of Deep Features and Image Labels for Large-scale Radiology Image Categorization and Scene Recognition
• Object Recognition with and without Objects
• A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?
• Effect of the output activation function on the probabilities and errors in medical image segmentation
• Why bigger is not always better: on finite and infinite neural networks
• Strength from Weakness: Fast Learning Using Weak Supervision
• CytoImageNet: A large-scale pretraining dataset for bioimage transfer learning
• Human perception in computer vision
• MosAIc: Finding Artistic Connections across Culture with Conditional Image Retrieval
• ImageNet-21K Pretraining for the Masses
• Curriculum By Smoothing
• Adversarial Training Helps Transfer Learning via Better Representations
• Interspecies Knowledge Transfer for Facial Keypoint Detection
• SB-MTL: Score-based Meta Transfer-Learning for Cross-Domain Few-Shot Learning
• Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning
• Material Classification using Neural Networks
• A Foliated View of Transfer Learning
• Approximate Cross-Validation for Structured Models
• An Investigation of Transfer Learning-Based Sentiment Analysis in Japanese
• Sequential modeling of Sessions using Recurrent Neural Networks for Skip Prediction
• Guided Generative Adversarial Neural Network for Representation Learning and High Fidelity Audio Generation using Fewer Labelled Audio Data
• Parameter Prediction for Unseen Deep Architectures
• Evaluating Self and Semi-Supervised Methods for Remote Sensing Segmentation Tasks
• Land Cover Mapping in Limited Labels Scenario: A Survey
• Uncovering the Connections Between Adversarial Transferability and Knowledge Transferability
• Understanding the Effects of Pre-Training for Object Detectors via Eigenspectrum
• Effective Domain Knowledge Transfer with Soft Fine-tuning
• Improving Automatic Jazz Melody Generation by Transfer Learning Techniques
• Pre-training without Natural Images
• Efficient Object Embedding for Spliced Image Retrieval
• Intraclass clustering: an implicit learning ability that regularizes DNNs
• Ship classification from overhead imagery using synthetic data and domain adaptation
• Transfer Learning with Human Corneal Tissues: An Analysis of Optimal Cut-Off Layer
• Hierarchical Transfer Convolutional Neural Networks for Image Classification
• Class Subset Selection for Transfer Learning using Submodularity
• Transfer Learning in Visual and Relational Reasoning
• DEPARA: Deep Attribution Graph for Deep Knowledge Transferability
• Meta-learning Transferable Representations with a Single Target Domain
• Multirate Training of Neural Networks
• Growing a Brain: Fine-Tuning by Increasing Model Capacity
• On The Transferability of Deep-Q Networks
• Shared Multi-Task Imitation Learning for Indoor Self-Navigation
• Clustering augmented Self-Supervised Learning: Anapplication to Land Cover Mapping
• Effects of Layer Freezing on Transferring a Speech Recognition System to Under-resourced Languages
• Neural Program Meta-Induction
• Semantic and Visual Similarities for Efficient Knowledge Transfer in CNN Training
• Beyond Flatland: Pre-training with a Strong 3D Inductive Bias
• Pay Attention to Convolution Filters: Towards Fast and Accurate Fine-Grained Transfer Learning
• A Self-Supervised Auxiliary Loss for Deep RL in Partially Observable Settings
• Measuring Information Transfer in Neural Networks
• Towards Making Deep Transfer Learning Never Hurt
• Focusing on the Big Picture: Insights into a Systems Approach to Deep Learning for Satellite Imagery
• Modular network for high accuracy object detection
• A Transferable Adaptive Domain Adversarial Neural Network for Virtual Reality Augmented EMG-Based Gesture Recognition