How transferable are features in deep neural networks?

Abstract

Many deep neural networks trained on natural images exhibit a curious phenomenon in common: on the first layer they learn features similar to Gabor filters and color blobs. Such first-layer features appear not to be specific to a particular dataset or task, but general in that they are applicable to many datasets and tasks. Features must eventually transition from general to specific by the last layer of the network, but this transition has not been studied extensively. In this paper we experimentally quantify the generality versus specificity of neurons in each layer of a deep convolutional neural network and report a few surprising results. Transferability is negatively affected by two distinct issues: (1) the specialization of higher layer neurons to their original task at the expense of performance on the target task, which was expected, and (2) optimization difficulties related to splitting networks between co-adapted neurons, which was not expected. In an example network trained on ImageNet, we demonstrate that either of these two issues may dominate, depending on whether features are transferred from the bottom, middle, or top of the network. We also document that the transferability of features decreases as the distance between the base task and target task increases, but that transferring features even from distant tasks can be better than using random features. A final surprising result is that initializing a network with transferred features from almost any number of layers can produce a boost to generalization that lingers even after fine-tuning to the target dataset.

To appear in Advances in Neural Information Processing Systems 27 (NIPS 2014)

References in corpus (1)

• Improving neural networks by preventing co-adaptation of feature detectors

Cited by in corpus (64)

• Learning Transferable Features with Deep Adaptation Networks
• DeepNAT: Deep Convolutional Neural Network for Segmenting Neuroanatomy
• Multi-source Transfer Learning with Convolutional Neural Networks for Lung Pattern Analysis
• Forecasting day-ahead electricity prices in Europe: the importance of considering market integration
• Automated Machine Learning: State-of-The-Art and Open Challenges
• Learning the PE Header, Malware Detection with Minimal Domain Knowledge
• Continual Learning via Neural Pruning
• POBA-GA: Perturbation Optimized Black-Box Adversarial Attacks via Genetic Algorithm
• Hermite-Gaussian Mode Detection via Convolution Neural Networks
• Deep Transfer Learning Methods for Colon Cancer Classification in Confocal Laser Microscopy Images
• A Selective Overview of Deep Learning
• FixyNN: Efficient Hardware for Mobile Computer Vision via Transfer Learning
• Learning to Transfer Examples for Partial Domain Adaptation
• 3-D Convolutional Neural Networks for Glioblastoma Segmentation
• Score Function Features for Discriminative Learning: Matrix and Tensor Framework
• Spatiotemporal Filtering for Event-Based Action Recognition
• Scaling Video Analytics on Constrained Edge Nodes
• Colorectal cancer diagnosis from histology images: A comparative study
• Factors of Transferability for a Generic ConvNet Representation
• Deep Learning for Design and Retrieval of Nano-photonic Structures
• Modeling Spatial and Temporal Cues for Multi-label Facial Action Unit Detection
• Why my photos look sideways or upside down? Detecting Canonical Orientation of Images using Convolutional Neural Networks
• Land Use and Land Cover Classification Using Deep Learning Techniques
• Transferring Autonomous Driving Knowledge on Simulated and Real Intersections
• UI-Net: Interactive Artificial Neural Networks for Iterative Image Segmentation Based on a User Model
• Deep Image Category Discovery using a Transferred Similarity Function
• Multi-Object Classification and Unsupervised Scene Understanding Using Deep Learning Features and Latent Tree Probabilistic Models
• How Transferable are CNN-based Features for Age and Gender Classification?
• Accelerating Deep Unsupervised Domain Adaptation with Transfer Channel Pruning
• Seismic data denoising and deblending using deep learning
• SuperTML: Two-Dimensional Word Embedding for the Precognition on Structured Tabular Data
• Embryo staging with weakly-supervised region selection and dynamically-decoded predictions
• Incremental Learning Using a Grow-and-Prune Paradigm with Efficient Neural Networks
• Differentially Private Continual Learning
• An Empirical Evaluation of Adversarial Robustness under Transfer Learning
• Detecting retail products in situ using CNN without human effort labeling
• Deep learning in bioinformatics: introduction, application, and perspective in big data era
• Tinkering Under the Hood: Interactive Zero-Shot Learning with Net Surgery
• Continual Learning with Self-Organizing Maps
• Detecting Colorized Images via Convolutional Neural Networks: Toward High Accuracy and Good Generalization
• Modelling Domain Relationships for Transfer Learning on Retrieval-based Question Answering Systems in E-commerce
• Dual Memory Architectures for Fast Deep Learning of Stream Data via an Online-Incremental-Transfer Strategy
• Can deep learning help you find the perfect match?
• Semantic Hierarchical Priors for Intrinsic Image Decomposition
• Sensitivity of Deep Convolutional Networks to Gabor Noise
• Learned Indexes for Dynamic Workloads
• Attention Transfer from Web Images for Video Recognition
• Distributed Deep Transfer Learning by Basic Probability Assignment
• Leveraging Socioeconomic Information and Deep Learning for Residential Load Pattern Prediction
• Few-Shot Learning-Based Human Activity Recognition
• Growing a Brain: Fine-Tuning by Increasing Model Capacity
• Understanding the Mechanism of Deep Learning Framework for Lesion Detection in Pathological Images with Breast Cancer
• Progressively Growing Generative Adversarial Networks for High Resolution Semantic Segmentation of Satellite Images
• High-Level Perceptual Similarity is Enabled by Learning Diverse Tasks
• Watermark retrieval from 3D printed objects via synthetic data training
• Unsupervised Domain Alignment to Mitigate Low Level Dataset Biases
• Learning Super-resolution 3D Segmentation of Plant Root MRI Images from Few Examples
• Integration of Text-maps in Convolutional Neural Networks for Region Detection among Different Textual Categories
• Network Implosion: Effective Model Compression for ResNets via Static Layer Pruning and Retraining
• EE-AE: An Exclusivity Enhanced Unsupervised Feature Learning Approach
• Combine PPO with NES to Improve Exploration
• Incorporating Human Domain Knowledge in 3D LiDAR-based Semantic Segmentation
• Learning to see across Domains and Modalities
• Sex-Prediction from Periocular Images across Multiple Sensors and Spectra