Bayesian Deep Learning and a Probabilistic Perspective of Generalization

Abstract

The key distinguishing property of a Bayesian approach is marginalization, rather than using a single setting of weights. Bayesian marginalization can particularly improve the accuracy and calibration of modern deep neural networks, which are typically underspecified by the data, and can represent many compelling but different solutions. We show that deep ensembles provide an effective mechanism for approximate Bayesian marginalization, and propose a related approach that further improves the predictive distribution by marginalizing within basins of attraction, without significant overhead. We also investigate the prior over functions implied by a vague distribution over neural network weights, explaining the generalization properties of such models from a probabilistic perspective. From this perspective, we explain results that have been presented as mysterious and distinct to neural network generalization, such as the ability to fit images with random labels, and show that these results can be reproduced with Gaussian processes. We also show that Bayesian model averaging alleviates double descent, resulting in monotonic performance improvements with increased flexibility. Finally, we provide a Bayesian perspective on tempering for calibrating predictive distributions.

arXiv comment31 pages, 19 figures

References in corpus (27)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• On Calibration of Modern Neural Networks
• Reconciling modern machine learning practice and the bias-variance trade-off
• Expectation Propagation for approximate Bayesian inference
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Weight Uncertainty in Neural Networks
• Understanding deep learning requires rethinking generalization
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
• Deep Ensembles: A Loss Landscape Perspective
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Essentially No Barriers in Neural Network Energy Landscape
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Fantastic Generalization Measures and Where to Find Them
• Functional Variational Bayesian Neural Networks
• The Case for Bayesian Deep Learning
• Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
• A Primer on PAC-Bayesian Learning
• Optimal Regularization Can Mitigate Double Descent
• The Functional Neural Process
• Understanding Generalization through Visualizations
• Safe-Bayesian Generalized Linear Regression
• Introducing an Explicit Symplectic Integration Scheme for Riemannian Manifold Hamiltonian Monte Carlo
• Output-Constrained Bayesian Neural Networks

Cited by in corpus (64)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• What Are Bayesian Neural Network Posteriors Really Like?
• Getting a CLUE: A Method for Explaining Uncertainty Estimates
• Finite Versus Infinite Neural Networks: an Empirical Study
• How Good is the Bayes Posterior in Deep Neural Networks Really?
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification
• Uncertainty Quantification and Deep Ensembles
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited
• Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
• Triple descent and the two kinds of overfitting: Where & why do they appear?
• Asymptotics of representation learning in finite Bayesian neural networks
• A Comparison of Uncertainty Estimation Approaches in Deep Learning Components for Autonomous Vehicle Applications
• BNNpriors: A library for Bayesian neural network inference with different prior distributions
• Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
• On Power Laws in Deep Ensembles
• Repulsive Deep Ensembles are Bayesian
• Learning under Model Misspecification: Applications to Variational and Ensemble methods
• A Quantitative Comparison of Epistemic Uncertainty Maps Applied to Multi-Class Segmentation
• A Simple Probabilistic Method for Deep Classification under Input-Dependent Label Noise
• Generalization bounds for deep learning
• Exact marginal prior distributions of finite Bayesian neural networks
• Bayesian active learning for production, a systematic study and a reusable library
• Leveraging Uncertainty for Improved Static Malware Detection Under Extreme False Positive Constraints
• Dimensionality reduction, regularization, and generalization in overparameterized regressions
• A Bayesian Perspective on Training Speed and Model Selection
• Scalable marginalization of correlated latent variables with applications to learning particle interaction kernels
• Deep transformation models for functional outcome prediction after acute ischemic stroke
• Is SGD a Bayesian sampler? Well, almost
• LiBRe: A Practical Bayesian Approach to Adversarial Detection
• Dangers of Bayesian Model Averaging under Covariate Shift
• Deep Ensembles from a Bayesian Perspective
• On Stein Variational Neural Network Ensembles
• Precise characterization of the prior predictive distribution of deep ReLU networks
• Walsh-Hadamard Variational Inference for Bayesian Deep Learning
• Training on Test Data with Bayesian Adaptation for Covariate Shift
• A benchmark study on reliable molecular supervised learning via Bayesian learning
• A comprehensive study on the prediction reliability of graph neural networks for virtual screening
• Simulation-Based Inference with Approximately Correct Parameters via Maximum Entropy
• Structured Weight Priors for Convolutional Neural Networks
• A Bayesian neural network predicts the dissolution of compact planetary systems
• An information-based metric for observing strategy optimization, demonstrated in the context of photometric redshifts with applications to cosmology
• Variational Auto-Regressive Gaussian Processes for Continual Learning
• Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect
• Structured Dropout Variational Inference for Bayesian Neural Networks
• Distributional Gaussian Process Layers for Outlier Detection in Image Segmentation
• Quantifying Uncertainty in Deep Spatiotemporal Forecasting
• Noisy Training Improves E2E ASR for the Edge
• Who's Afraid of Thomas Bayes?
• Identifying and Exploiting Structures for Reliable Deep Learning
• Contributions to Large Scale Bayesian Inference and Adversarial Machine Learning
• Intrinsic uncertainties and where to find them
• Calibration and Uncertainty Quantification of Bayesian Convolutional Neural Networks for Geophysical Applications
• Pathologies in priors and inference for Bayesian transformers
• The Effect of Prior Lipschitz Continuity on the Adversarial Robustness of Bayesian Neural Networks
• Improving Uncertainty Calibration via Prior Augmented Data
• Posterior Temperature Optimization in Variational Inference for Inverse Problems
• Rapid Risk Minimization with Bayesian Models Through Deep Learning Approximation
• Greedy Bayesian Posterior Approximation with Deep Ensembles
• Self-Reflective Variational Autoencoder