Distributional Generalization: A New Kind of Generalization

Abstract

We introduce a new notion of generalization -- Distributional Generalization -- which roughly states that outputs of a classifier at train and test time are close *as distributions*, as opposed to close in just their average error. For example, if we mislabel 30% of dogs as cats in the train set of CIFAR-10, then a ResNet trained to interpolation will in fact mislabel roughly 30% of dogs as cats on the *test set* as well, while leaving other classes unaffected. This behavior is not captured by classical generalization, which would only consider the average error and not the distribution of errors over the input domain. Our formal conjectures, which are much more general than this example, characterize the form of distributional generalization that can be expected in terms of problem parameters: model architecture, training procedure, number of samples, and data distribution. We give empirical evidence for these conjectures across a variety of domains in machine learning, including neural networks, kernel machines, and decision trees. Our results thus advance our empirical understanding of interpolating classifiers.

arXiv commentCo-first authors. V2: Intro shortened; no new results

References in corpus (31)

• Scikit-learn: Machine Learning in Python
• The NumPy array: a structure for efficient numerical computation
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Wide Residual Networks
• On Calibration of Modern Neural Networks
• Reconciling modern machine learning practice and the bias-variance trade-off
• Benign Overfitting in Linear Regression
• Deep Learning is Robust to Massive Label Noise
• Understanding the Role of Individual Units in a Deep Neural Network
• Deep Ensembles: A Loss Landscape Perspective
• Learning to Generate Reviews and Discovering Sentiment
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Uniform convergence may be unable to explain generalization in deep learning
• Combating Label Noise in Deep Learning Using Abstention
• To understand deep learning we need to understand kernel learning
• Generalisation error in learning with random features and the hidden manifold model
• Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks
• A Modern Take on the Bias-Variance Tradeoff in Neural Networks
• Stiffness: A New Perspective on Generalization in Neural Networks
• Neural Kernels Without Tangents
• Making Sense of Random Forest Probabilities: a Kernel Perspective
• A Neural Scaling Law from the Dimension of the Data Manifold
• Learning Not to Learn in the Presence of Noisy Labels
• RFCDE: Random Forests for Conditional Density Estimation
• Generalisation dynamics of online learning in over-parameterised neural networks
• Physical Attribute Prediction Using Deep Residual Neural Networks
• Wider Networks Learn Better Features

Cited by in corpus (6)

• Explaining Neural Scaling Laws
• An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels
• Revisiting Model Stitching to Compare Neural Representations
• Assessing Generalization of SGD via Disagreement
• The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers
• Explaining generalization in deep learning: progress and fundamental limits