Why flatness does and does not correlate with generalization for deep neural networks
arXiv:2103.06219
Abstract
The intuition that local flatness of the loss landscape is correlated with better generalization for deep neural networks (DNNs) has been explored for decades, spawning many different flatness measures. Recently, this link with generalization has been called into question by a demonstration that many measures of flatness are vulnerable to parameter re-scaling which arbitrarily changes their value without changing neural network outputs. Here we show that, in addition, some popular variants of SGD such as Adam and Entropy-SGD, can also break the flatness-generalization correlation. As an alternative to flatness measures, we use a function based picture and propose using the log of Bayesian prior upon initialization, , as a predictor of the generalization when a DNN converges on function after training to zero error. The prior is directly proportional to the Bayesian posterior for functions that give zero error on a test set. For the case of image classification, we show that is a significantly more robust predictor of generalization than flatness measures are. Whilst local flatness measures fail under parameter re-scaling, the prior/posterior, which is global quantity, remains invariant under re-scaling. Moreover, the correlation with generalization as a function of data complexity remains good for different variants of SGD.
References in corpus (33)
- Adam: A Method for Stochastic Optimization
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Understanding deep learning requires rethinking generalization
- Don't Decay the Learning Rate, Increase the Batch Size
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- Deep Neural Networks as Gaussian Processes
- Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
- A Closer Look at Memorization in Deep Networks
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- Sensitivity and Generalization in Neural Networks: an Empirical Study
- Averaging Weights Leads to Wider Optima and Better Generalization
- Fantastic Generalization Measures and Where to Find Them
- Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
- Sharp Minima Can Generalize For Deep Nets
- Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
- Predicting the Generalization Gap in Deep Networks with Margin Distributions
- Deep learning generalizes because the parameter-function map is biased towards simple functions
- From genotypes to organisms: State-of-the-art and perspectives of a cornerstone in evolutionary dynamics
- The arrival of the frequent: how bias in genotype-phenotype maps can steer populations to local optima
- Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
- Hessian-based Analysis of Large Batch Training and Robustness to Adversaries
- Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning
- A Fine-Grained Spectral Perspective on Neural Networks
- Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited
- A Surprising Linear Relationship Predicts Test Performance in Deep Networks
- A Scale Invariant Flatness Measure for Deep Network Minima
- Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysis
- Generalization bounds for deep learning
- Neural networks are a priori biased towards Boolean functions with low entropy
- How noise affects the Hessian spectrum in overparameterized neural networks
- Learning by Turning: Neural Architecture Aware Optimisation
- In Search of Robust Measures of Generalization
- A Reparameterization-Invariant Flatness Measure for Deep Neural Networks