Is SGD a Bayesian sampler? Well, almost

Abstract

Overparameterised deep neural networks (DNNs) are highly expressive and so can, in principle, generate almost any function that fits a training dataset with zero error. The vast majority of these functions will perform poorly on unseen data, and yet in practice DNNs often generalise remarkably well. This success suggests that a trained DNN must have a strong inductive bias towards functions with low generalisation error. Here we empirically investigate this inductive bias by calculating, for a range of architectures and datasets, the probability $P_{SGD}(f\mid S)$ that an overparameterised DNN, trained with stochastic gradient descent (SGD) or one of its variants, converges on a function $f$ consistent with a training set $S$. We also use Gaussian processes to estimate the Bayesian posterior probability $P_B(f\mid S)$ that the DNN expresses $f$ upon random sampling of its parameters, conditioned on $S$. Our main findings are that $P_{SGD}(f\mid S)$ correlates remarkably well with $P_B(f\mid S)$ and that $P_B(f\mid S)$ is strongly biased towards low-error and low complexity functions. These results imply that strong inductive bias in the parameter-function map (which determines $P_B(f\mid S)$), rather than a special property of SGD, is the primary explanation for why DNNs generalise so well in the overparameterised regime. While our results suggest that the Bayesian posterior $P_B(f\mid S)$ is the first order determinant of $P_{SGD}(f\mid S)$, there remain second order differences that are sensitive to hyperparameter tuning. A function probability picture, based on $P_{SGD}(f\mid S)$ and/or $P_B(f\mid S)$, can shed new light on the way that variations in architecture or hyperparameter settings such as batch size, learning rate, and optimiser choice, affect DNN performance.

References in corpus (36)

• Deep Learning in Neural Networks: An Overview
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• ADADELTA: An Adaptive Learning Rate Method
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Scaling Laws for Neural Language Models
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Opening the Black Box of Deep Neural Networks via Information
• The Loss Surfaces of Multilayer Networks
• Don't Decay the Learning Rate, Increase the Batch Size
• Deep Neural Networks as Gaussian Processes
• Deep Learning Scaling is Predictable, Empirically
• On Exact Computation with an Infinitely Wide Neural Net
• A Closer Look at Memorization in Deep Networks
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Exponential expressivity in deep neural networks through transient chaos
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• On Empirical Comparisons of Optimizers for Deep Learning
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Fantastic Generalization Measures and Where to Find Them
• On the Spectral Bias of Neural Networks
• Sharp Minima Can Generalize For Deep Nets
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• The arrival of the frequent: how bias in genotype-phenotype maps can steer populations to local optima
• Perspective: Energy Landscapes for Machine Learning
• A Philosophical Treatise of Universal Induction
• Power-law distributions for the areas of the basins of attraction on a potential energy landscape
• Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning
• The promises and pitfalls of Stochastic Gradient Langevin Dynamics
• A Fine-Grained Spectral Perspective on Neural Networks
• Random deep neural networks are biased towards simple functions
• Geometry of energy landscapes and the optimizability of deep neural networks
• Neural networks are a priori biased towards Boolean functions with low entropy
• How noise affects the Hessian spectrum in overparameterized neural networks
• Can Implicit Bias Explain Generalization? Stochastic Convex Optimization as a Case Study

Cited by in corpus (2)

• Efficient Bayesian Uncertainty Estimation for nnU-Net
• A Bayesian neural network predicts the dissolution of compact planetary systems