Law of Large Numbers for Bayesian two-layer Neural Network trained with Variational Inference

Abstract

We provide a rigorous analysis of training by variational inference (VI) of Bayesian neural networks in the two-layer and infinite-width case. We consider a regression problem with a regularized evidence lower bound (ELBO) which is decomposed into the expected log-likelihood of the data and the Kullback-Leibler (KL) divergence between the a priori distribution and the variational posterior. With an appropriate weighting of the KL, we prove a law of large numbers for three different training schemes: (i) the idealized case with exact estimation of a multiple Gaussian integral from the reparametrization trick, (ii) a minibatch scheme using Monte Carlo sampling, commonly known as Bayes by Backprop, and (iii) a new and computationally cheaper algorithm which we introduce as Minimal VI. An important result is that all methods converge to the same mean-field limit. Finally, we illustrate our results numerically and discuss the need for the derivation of a central limit theorem.

References in corpus (17)

• Auto-Encoding Variational Bayes
• Variational Inference: A Review for Statisticians
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• What Are Bayesian Neural Network Posteriors Really Like?
• On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
• From the master equation to mean field game limit theory: A central limit theorem
• A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks
• Quantitative Propagation of Chaos for SGD in Wide Neural Networks
• Infinite-width limit of deep linear neural networks
• Wide Mean-Field Bayesian Neural Networks Ignore the Data
• Variational Inference of overparameterized Bayesian Neural Networks: a theoretical and empirical study
• Law of large numbers and central limit theorem for wide two-layer neural networks: the mini-batch and noisy case