Why Do Better Loss Functions Lead to Less Transferable Features?
arXiv:2010.16402
Abstract
Previous work has proposed many new loss functions and regularizers that improve test accuracy on image classification tasks. However, it is not clear whether these loss functions learn better representations for downstream tasks. This paper studies how the choice of training objective affects the transferability of the hidden representations of convolutional neural networks trained on ImageNet. We show that many objectives lead to statistically significant improvements in ImageNet accuracy over vanilla softmax cross-entropy, but the resulting fixed feature extractors transfer substantially worse to downstream tasks, and the choice of loss has little effect when networks are fully fine-tuned on the new tasks. Using centered kernel alignment to measure similarity between hidden representations of networks, we find that differences among loss functions are apparent only in the last few layers of the network. We delve deeper into representations of the penultimate layer, finding that different objectives and hyperparameter combinations lead to dramatically different levels of class separation. Representations with higher class separation obtain higher accuracy on the original task, but their features are less useful for downstream tasks. Our results suggest there exists a trade-off between learning invariant features for the original task and features relevant for transfer tasks.
arXiv commentNeurIPS 2021
References in corpus (27)
- Learning Transferable Visual Models From Natural Language Supervision
- Prototypical Networks for Few-shot Learning
- DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
- How transferable are features in deep neural networks?
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- On Calibration of Modern Neural Networks
- Opening the Black Box of Deep Neural Networks via Information
- Similarity of Neural Network Representations Revisited
- Do ImageNet Classifiers Generalize to ImageNet?
- Algorithms for Learning Kernels Based on Centered Alignment
- Regularizing Neural Networks by Penalizing Confident Output Distributions
- Prevalence of Neural Collapse during the terminal phase of deep learning training
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- Learning Robust Global Representations by Penalizing Local Predictive Power
- Dropout Training as Adaptive Regularization
- Natural Adversarial Examples
- Intrinsic dimension of data representations in deep neural networks
- Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
- Norm-Based Capacity Control in Neural Networks
- Are we done with ImageNet?
- Exploring the Limits of Large Scale Pre-training
- Analyzing and Improving Representations with the Soft Nearest Neighbor Loss
- Towards Understanding Label Smoothing
- Altitude Training: Strong Bounds for Single-Layer Dropout
- Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks
- On Convergence and Generalization of Dropout Training
- An Investigation of how Label Smoothing Affects Generalization