Provable Smoothness Guarantees for Black-Box Variational Inference
arXiv:1901.08431
Abstract
Black-box variational inference tries to approximate a complex target distribution though a gradient-based optimization of the parameters of a simpler distribution. Provable convergence guarantees require structural properties of the objective. This paper shows that for location-scale family approximations, if the target is M-Lipschitz smooth, then so is the objective, if the entropy is excluded. The key proof idea is to describe gradients in a certain inner-product space, thus permitting use of Bessel's inequality. This result gives insight into how to parameterize distributions, gives bounds the location of the optimal parameters, and is a key ingredient for convergence guarantees.
arXiv commentInternational Conference on Machine Learning (ICML) 2020
References in corpus (10)
- Auto-Encoding Variational Bayes
- Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
- Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization
- Yes, but Did It Work?: Evaluating Variational Inference
- Automated Variational Inference in Probabilistic Programming
- Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models
- Boosting Variational Inference
- Boosting Variational Inference: an Optimization Perspective
- Fast Second-Order Stochastic Backpropagation for Variational Inference
- Provable Gradient Variance Guarantees for Black-Box Variational Inference