How transferable are features in deep neural networks?

Abstract

Many deep neural networks trained on natural images exhibit a curious phenomenon in common: on the first layer they learn features similar to Gabor filters and color blobs. Such first-layer features appear not to be specific to a particular dataset or task, but general in that they are applicable to many datasets and tasks. Features must eventually transition from general to specific by the last layer of the network, but this transition has not been studied extensively. In this paper we experimentally quantify the generality versus specificity of neurons in each layer of a deep convolutional neural network and report a few surprising results. Transferability is negatively affected by two distinct issues: (1) the specialization of higher layer neurons to their original task at the expense of performance on the target task, which was expected, and (2) optimization difficulties related to splitting networks between co-adapted neurons, which was not expected. In an example network trained on ImageNet, we demonstrate that either of these two issues may dominate, depending on whether features are transferred from the bottom, middle, or top of the network. We also document that the transferability of features decreases as the distance between the base task and target task increases, but that transferring features even from distant tasks can be better than using random features. A final surprising result is that initializing a network with transferred features from almost any number of layers can produce a boost to generalization that lingers even after fine-tuning to the target dataset.

To appear in Advances in Neural Information Processing Systems 27 (NIPS 2014)

References in corpus (1)

• Improving neural networks by preventing co-adaptation of feature detectors

Cited by in corpus (141)

• Learning Transferable Features with Deep Adaptation Networks
• DeepNAT: Deep Convolutional Neural Network for Segmenting Neuroanatomy
• Multi-source Transfer Learning with Convolutional Neural Networks for Lung Pattern Analysis
• Forecasting day-ahead electricity prices in Europe: the importance of considering market integration
• How Does Learning Rate Decay Help Modern Neural Networks?
• Automated Machine Learning: State-of-The-Art and Open Challenges
• Learning the PE Header, Malware Detection with Minimal Domain Knowledge
• Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs
• Continual Learning via Neural Pruning
• POBA-GA: Perturbation Optimized Black-Box Adversarial Attacks via Genetic Algorithm
• Feature-Critic Networks for Heterogeneous Domain Generalization
• Learning to Continually Learn
• Rethinking the Hyperparameters for Fine-tuning
• Multi-modal Deep Analysis for Multimedia
• Arabic Handwritten Character Recognition based on Convolution Neural Networks and Support Vector Machine
• Hermite-Gaussian Mode Detection via Convolution Neural Networks
• Dynamic Prediction of ICU Mortality Risk Using Domain Adaptation
• Deep Transfer Learning Methods for Colon Cancer Classification in Confocal Laser Microscopy Images
• PnP-AdaNet: Plug-and-Play Adversarial Domain Adaptation Network with a Benchmark at Cross-modality Cardiac Segmentation
• Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis
• A Selective Overview of Deep Learning
• FixyNN: Efficient Hardware for Mobile Computer Vision via Transfer Learning
• Learning to Transfer Examples for Partial Domain Adaptation
• In-domain representation learning for remote sensing
• 3-D Convolutional Neural Networks for Glioblastoma Segmentation
• Small-GAN: Speeding Up GAN Training Using Core-sets
• Regularizing Deep Multi-Task Networks using Orthogonal Gradients
• Drop to Adapt: Learning Discriminative Features for Unsupervised Domain Adaptation
• Score Function Features for Discriminative Learning: Matrix and Tensor Framework
• Spatiotemporal Filtering for Event-Based Action Recognition
• Scaling Video Analytics on Constrained Edge Nodes
• Scalable Transfer Learning with Expert Models
• RC-DARTS: Resource Constrained Differentiable Architecture Search
• Factors of Transferability for a Generic ConvNet Representation
• FEED: Feature-level Ensemble for Knowledge Distillation
• Colorectal cancer diagnosis from histology images: A comparative study
• Locality and compositionality in zero-shot learning
• On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation
• Deep Learning for Design and Retrieval of Nano-photonic Structures
• Modeling Spatial and Temporal Cues for Multi-label Facial Action Unit Detection
• Revisiting Meta-Learning as Supervised Learning
• Virtual-to-Real-World Transfer Learning for Robots on Wilderness Trails
• When Semi-Supervised Learning Meets Transfer Learning: Training Strategies, Models and Datasets
• RoboCup 2019 AdultSize Winner NimbRo: Deep Learning Perception, In-Walk Kick, Push Recovery, and Team Play Capabilities
• Why my photos look sideways or upside down? Detecting Canonical Orientation of Images using Convolutional Neural Networks
• Robust Watermarking of Neural Network with Exponential Weighting
• Land Use and Land Cover Classification Using Deep Learning Techniques
• Transferring Autonomous Driving Knowledge on Simulated and Real Intersections
• The effect of task and training on intermediate representations in convolutional neural networks revealed with modified RV similarity analysis
• UI-Net: Interactive Artificial Neural Networks for Iterative Image Segmentation Based on a User Model
• A Comprehensive Analysis of Information Leakage in Deep Transfer Learning
• Multi-Object Classification and Unsupervised Scene Understanding Using Deep Learning Features and Latent Tree Probabilistic Models
• Deep Image Category Discovery using a Transferred Similarity Function
• Toward Automatic Threat Recognition for Airport X-ray Baggage Screening with Deep Convolutional Object Detection
• How Transferable are CNN-based Features for Age and Gender Classification?
• SuperTML: Two-Dimensional Word Embedding for the Precognition on Structured Tabular Data
• Retrieving Similar E-Commerce Images Using Deep Learning
• Accelerating Deep Unsupervised Domain Adaptation with Transfer Channel Pruning
• Neural network augmented wave-equation simulation
• Seismic data denoising and deblending using deep learning
• Incremental Learning Using a Grow-and-Prune Paradigm with Efficient Neural Networks
• To Transfer or Not to Transfer: Misclassification Attacks Against Transfer Learned Text Classifiers
• Structured Pruning for Efficient ConvNets via Incremental Regularization
• FSD-10: A Dataset for Competitive Sports Content Analysis
• AdapNet: Adaptability Decomposing Encoder-Decoder Network for Weakly Supervised Action Recognition and Localization
• Embryo staging with weakly-supervised region selection and dynamically-decoded predictions
• Adversarial Domain Adaptation with Domain Mixup
• Deep causal representation learning for unsupervised domain adaptation
• Differentially Private Continual Learning
• Unsupervised Transfer Learning via BERT Neuron Selection
• Actor Critic with Differentially Private Critic
• Disentangled Cumulants Help Successor Representations Transfer to New Tasks
• Classifying the classifier: dissecting the weight space of neural networks
• An Empirical Evaluation of Adversarial Robustness under Transfer Learning
• Deep Learning and Control Algorithms of Direct Perception for Autonomous Driving
• Deep learning in bioinformatics: introduction, application, and perspective in big data era
• Improving Robustness and Generality of NLP Models Using Disentangled Representations
• Towards Neural Network Patching: Evaluating Engagement-Layers and Patch-Architectures
• Guidelines and Benchmarks for Deployment of Deep Learning Models on Smartphones as Real-Time Apps
• Detecting retail products in situ using CNN without human effort labeling
• Discriminative Adversarial Domain Adaptation
• SimEx: Express Prediction of Inter-dataset Similarity by a Fleet of Autoencoders
• Adaptation of a deep learning malignancy model from full-field digital mammography to digital breast tomosynthesis
• Human-centric Transfer Learning Explanation via Knowledge Graph [Extended Abstract]
• Tinkering Under the Hood: Interactive Zero-Shot Learning with Net Surgery
• Finding Quantum Critical Points with Neural-Network Quantum States
• Continual Learning with Self-Organizing Maps
• Detecting Colorized Images via Convolutional Neural Networks: Toward High Accuracy and Good Generalization
• Obstruction level detection of sewer videos using convolutional neural networks
• Can deep learning help you find the perfect match?
• Dual Memory Architectures for Fast Deep Learning of Stream Data via an Online-Incremental-Transfer Strategy
• Modelling Domain Relationships for Transfer Learning on Retrieval-based Question Answering Systems in E-commerce
• Distributed Deep Transfer Learning by Basic Probability Assignment
• Attention Transfer from Web Images for Video Recognition
• Sensitivity of Deep Convolutional Networks to Gabor Noise
• Learned Indexes for Dynamic Workloads
• Semantic Hierarchical Priors for Intrinsic Image Decomposition
• Transfer learning from language models to image caption generators: Better models may not transfer better
• Frosting Weights for Better Continual Training
• Lightweight and Robust Representation of Economic Scales from Satellite Imagery
• An Attention-Based Speaker Naming Method for Online Adaptation in Non-Fixed Scenarios
• Few-Shot Learning-Based Human Activity Recognition
• Multi-Task Feature Learning for Knowledge Graph Enhanced Recommendation
• Ward2ICU: A Vital Signs Dataset of Inpatients from the General Ward
• Transfer Learning in 4D for Breast Cancer Diagnosis using Dynamic Contrast-Enhanced Magnetic Resonance Imaging
• Accelerating Multi-Model Inference by Merging DNNs of Different Weights
• Domain Adaption for Knowledge Tracing
• Leveraging Socioeconomic Information and Deep Learning for Residential Load Pattern Prediction
• Transfer Learning in General Lensless Imaging through Scattering Media
• Machine Learning Techniques to Detect and Characterise Whistler Radio Waves
• Parallelized Interactive Machine Learning on Autonomous Vehicles
• Understanding the Mechanism of Deep Learning Framework for Lesion Detection in Pathological Images with Breast Cancer
• Growing a Brain: Fine-Tuning by Increasing Model Capacity
• High-Level Perceptual Similarity is Enabled by Learning Diverse Tasks
• Unsupervised Domain Alignment to Mitigate Low Level Dataset Biases
• Progressively Growing Generative Adversarial Networks for High Resolution Semantic Segmentation of Satellite Images
• Neural Crossbreed: Neural Based Image Metamorphosis
• Fused Deep Convolutional Neural Network for Precision Diagnosis of COVID-19 Using Chest X-Ray Images
• Program synthesis performance constrained by non-linear spatial relations in Synthetic Visual Reasoning Test
• Watermark retrieval from 3D printed objects via synthetic data training
• Shareable Representations for Search Query Understanding
• Combine PPO with NES to Improve Exploration
• Diabetic Retinopathy detection by retinal image recognizing
• 3D Object Segmentation for Shelf Bin Picking by Humanoid with Deep Learning and Occupancy Voxel Grid Map
• Residual Continual Learning
• Granular Motor State Monitoring of Free Living Parkinson's Disease Patients via Deep Learning
• Fine tuning U-Net for ultrasound image segmentation: which layers?
• Low-Cost Transfer Learning of Face Tasks
• Model Adaption Object Detection System for Robot
• A Deep Learning Approach for Robust Corridor Following
• Learning to see across Domains and Modalities
• Learning Super-resolution 3D Segmentation of Plant Root MRI Images from Few Examples
• Transferable End-to-End Aspect-based Sentiment Analysis with Selective Adversarial Learning
• EE-AE: An Exclusivity Enhanced Unsupervised Feature Learning Approach
• Sex-Prediction from Periocular Images across Multiple Sensors and Spectra
• Learning to Generalize One Sample at a Time with Self-Supervision
• IIITM Face: A Database for Facial Attribute Detection in Constrained and Simulated Unconstrained Environments
• Incorporating Human Domain Knowledge in 3D LiDAR-based Semantic Segmentation
• DSRGAN: Explicitly Learning Disentangled Representation of Underlying Structure and Rendering for Image Generation without Tuple Supervision
• Integration of Text-maps in Convolutional Neural Networks for Region Detection among Different Textual Categories
• Network Implosion: Effective Model Compression for ResNets via Static Layer Pruning and Retraining