Convergence Rates of Variational Inference in Sparse Deep Learning

Abstract

Variational inference is becoming more and more popular for approximating intractable posterior distributions in Bayesian statistics and machine learning. Meanwhile, a few recent works have provided theoretical justification and new insights on deep neural networks for estimating smooth functions in usual settings such as nonparametric regression. In this paper, we show that variational inference for sparse deep learning retains the same generalization properties than exact Bayesian inference. In particular, we highlight the connection between estimation and approximation theories via the classical bias-variance trade-off and show that it leads to near-minimax rates of convergence for Hölder smooth functions. Additionally, we show that the model selection framework over the neural network architecture via ELBO maximization does not overfit and adaptively achieves the optimal rate of convergence.

References in corpus (2)

• Expectation Propagation for approximate Bayesian inference
• Bayesian fractional posteriors

Cited by in corpus (6)

• MMD-Bayes: Robust Bayesian Estimation via Maximum Mean Discrepancy
• Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details
• Adaptive variational Bayes: Optimality, computation and applications
• Misclassification bounds for PAC-Bayesian sparse deep learning
• Hidden Markov Neural Networks
• Adaptive Variational Bayesian Inference for Sparse Deep Neural Network