Fantastic Generalization Measures and Where to Find Them

Abstract

Generalization of deep networks has been of great interest in recent years, resulting in a number of theoretically and empirically motivated complexity measures. However, most papers proposing such measures study only a small set of models, leaving open the question of whether the conclusion drawn from those experiments would remain valid in other settings. We present the first large scale study of generalization in deep networks. We investigate more then 40 complexity measures taken from both theoretical bounds and empirical studies. We train over 10,000 convolutional networks by systematically varying commonly used hyperparameters. Hoping to uncover potentially causal relationships between each measure and generalization, we analyze carefully controlled experiments and show surprising failures of some measures as well as promising measures for further research.

References in corpus (16)

• Robustness of a Network of Networks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Regularizing and Optimizing LSTM Language Models
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Sharp Minima Can Generalize For Deep Nets
• Predicting the Generalization Gap in Deep Networks with Margin Distributions
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• Generalization in Deep Networks: The Role of Distance from Initialization
• Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience
• Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
• PAC-Bayesian Margin Bounds for Convolutional Neural Networks

Cited by in corpus (51)

• The effect of data encoding on the expressive power of variational quantum machine learning models
• What is being transferred in transfer learning?
• Evaluation of the MACE Force Field Architecture: from Medicinal Chemistry to Materials Science
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• The Modern Mathematics of Deep Learning
• Sharpness-Aware Minimization for Efficiently Improving Generalization
• Understanding quantum machine learning also requires rethinking generalization
• The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning
• Analysis of Generalizability of Deep Neural Networks Based on the Complexity of Decision Boundary
• Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited
• Entropic gradient descent algorithms and wide flat minima
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks
• Data efficiency and extrapolation trends in neural network interatomic potentials
• Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path
• The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers
• Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization
• Inductive Biases and Variable Creation in Self-Attention Mechanisms
• Empirical Risk Minimization with Relative Entropy Regularization
• Deep learning via message passing algorithms based on belief propagation
• Generalization bounds for deep learning
• Wide flat minima and optimal generalization in classifying high-dimensional Gaussian mixtures
• Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
• Is SGD a Bayesian sampler? Well, almost
• CNNs Avoid Curse of Dimensionality by Learning on Patches
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• Deep ReLU Networks Preserve Expected Length
• Perturbated Gradients Updating within Unit Space for Deep Learning
• Stochastic Training is Not Necessary for Generalization
• Neural Complexity Measures
• A Theoretical Analysis of Fine-tuning with Linear Teachers
• Enhancing Transformers without Self-supervised Learning: A Loss Landscape Perspective in Sequential Recommendation
• SWAD: Domain Generalization by Seeking Flat Minima
• DCI-ES: An Extended Disentanglement Framework with Connections to Identifiability
• Neural Network Complexity of Chaos and Turbulence
• FlatNAS: optimizing Flatness in Neural Architecture Search for Out-of-Distribution Robustness
• Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics
• Why flatness does and does not correlate with generalization for deep neural networks
• Implicit bias of deep linear networks in the large learning rate phase
• Optimal Machine Intelligence at the Edge of Chaos
• Flatness is a False Friend
• Compressing Heavy-Tailed Weight Matrices for Non-Vacuous Generalization Bounds
• Towards a practical measure of interference for reinforcement learning
• Formal limitations of sample-wise information-theoretic generalization bounds
• Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models
• Practical Assessment of Generalization Performance Robustness for Deep Networks via Contrastive Examples
• Generalized Resubstitution for Classification Error Estimation
• On the Robustness and Generalization of Deep Learning Driven Full Waveform Inversion
• Phases of learning dynamics in artificial neural networks: with or without mislabeled data
• Ridgeless Interpolation with Shallow ReLU Networks in $1D$ is Nearest Neighbor Curvature Extrapolation and Provably Generalizes on Lipschitz Functions
• Minimum sharpness: Scale-invariant parameter-robustness of neural networks
• Exact Stochastic Second Order Deep Learning