Convergence Rates of Variational Inference in Sparse Deep Learning
arXiv:1908.04847
Abstract
Variational inference is becoming more and more popular for approximating intractable posterior distributions in Bayesian statistics and machine learning. Meanwhile, a few recent works have provided theoretical justification and new insights on deep neural networks for estimating smooth functions in usual settings such as nonparametric regression. In this paper, we show that variational inference for sparse deep learning retains the same generalization properties than exact Bayesian inference. In particular, we highlight the connection between estimation and approximation theories via the classical bias-variance trade-off and show that it leads to near-minimax rates of convergence for Hölder smooth functions. Additionally, we show that the model selection framework over the neural network architecture via ELBO maximization does not overfit and adaptively achieves the optimal rate of convergence.
References in corpus (22)
- Expectation Propagation for approximate Bayesian inference
- A Convergence Theory for Deep Learning via Over-Parameterization
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- Frequentist Consistency of Variational Bayes
- Deep Rewiring: Training very sparse deep networks
- No bad local minima: Data independent training error guarantees for multilayer neural networks
- Practical Deep Learning with Bayesian Principles
- The power of deeper networks for expressing natural functions
- Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality
- On the loss landscape of a class of deep neural networks with no bad local valleys
- A Primer on PAC-Bayesian Learning
- Consistency of Variational Bayes Inference for Estimation and Model Selection in Mixtures
- Effect of Depth and Width on Local Minima in Deep Learning
- Understanding Priors in Bayesian Neural Networks at the Unit Level
- On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces
- SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient
- Posterior Concentration for Sparse Deep Learning
- Practical bounds on the error of Bayesian posterior approximations: A nonasymptotic approach
- Consistency of ELBO maximization for model selection
- A Generalization Bound for Online Variational Inference
- Bayesian fractional posteriors
- Asymptotic Consistency of Rényi-Approximate Posteriors
Cited by in corpus (6)
- MMD-Bayes: Robust Bayesian Estimation via Maximum Mean Discrepancy
- Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details
- Adaptive variational Bayes: Optimality, computation and applications
- Misclassification bounds for PAC-Bayesian sparse deep learning
- Hidden Markov Neural Networks
- Adaptive Variational Bayesian Inference for Sparse Deep Neural Network