What shapes feature representations? Exploring datasets, architectures, and training

Abstract

In naturalistic learning problems, a model's input contains a wide range of features, some useful for the task at hand, and others not. Of the useful features, which ones does the model use? Of the task-irrelevant features, which ones does the model represent? Answers to these questions are important for understanding the basis of models' decisions, as well as for building models that learn versatile, adaptable representations useful beyond the original training task. We study these questions using synthetic datasets in which the task-relevance of input features can be controlled directly. We find that when two features redundantly predict the labels, the model preferentially represents one, and its preference reflects what was most linearly decodable from the untrained model. Over training, task-relevant features are enhanced, and task-irrelevant features are partially suppressed. Interestingly, in some cases, an easier, weakly predictive feature can suppress a more strongly predictive, but more difficult one. Additionally, models trained to recognize both easy and hard features learn representations most similar to models that use only the easy feature. Further, easy features lead to more consistent representations across model runs than do hard features. Finally, models have greater representational similarity to an untrained model than to models trained on a different task. Our results highlight the complex processes that determine which features a model represents.

arXiv comment22 pages

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Overcoming catastrophic forgetting in neural networks
• Shortcut Learning in Deep Neural Networks
• Reconciling modern machine learning practice and the bias-variance trade-off
• Continual Learning Through Synaptic Intelligence
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Opening the Black Box of Deep Neural Networks via Information
• Similarity of Neural Network Representations Revisited
• What makes ImageNet good for transfer learning?
• A mathematical theory of semantic development in deep neural networks
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• The Pitfalls of Simplicity Bias in Neural Networks
• Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
• Learning explanations that are hard to vary
• An analytic theory of generalization dynamics and transfer learning in deep linear networks

Cited by in corpus (13)

• Shortcut Learning in Deep Neural Networks
• Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
• Intriguing Properties of Contrastive Losses
• Understanding the Failure Modes of Out-of-Distribution Generalization
• Gradient Starvation: A Learning Proclivity in Neural Networks
• Convolutional Neural Networks Rarely Learn Shape for Semantic Segmentation
• Dataset Distillation with Infinitely Wide Convolutional Networks
• Object Detector Differences when using Synthetic and Real Training Data
• Transforming task representations to perform novel tasks
• Saliency is a Possible Red Herring When Diagnosing Poor Generalization
• Classification of integers based on residue classes via modern deep learning algorithms
• Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)
• Learning Less Generalizable Patterns with an Asymmetrically Trained Double Classifier for Better Test-Time Adaptation