Robust, Accurate Stochastic Optimization for Variational Inference

Abstract

We consider the problem of fitting variational posterior approximations using stochastic optimization methods. The performance of these approximations depends on (1) how well the variational family matches the true posterior distribution,(2) the choice of divergence, and (3) the optimization of the variational objective. We show that even in the best-case scenario when the exact posterior belongs to the assumed variational family, common stochastic optimization methods lead to poor variational approximations if the problem dimension is moderately large. We also demonstrate that these methods are not robust across diverse model types. Motivated by these findings, we develop a more robust and accurate stochastic optimization framework by viewing the underlying optimization algorithm as producing a Markov chain. Our approach is theoretically motivated and includes a diagnostic for convergence and a novel stopping rule, both of which are robust to noisy evaluations of the objective function. We show empirically that the proposed framework works well on a diverse set of models: it can automatically detect stochastic optimization failure or inaccurate variational approximation

References in corpus (13)

• A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
• Pareto Smoothed Importance Sampling
• Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Frequentist Consistency of Variational Bayes
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Parallelizing Stochastic Gradient Descent for Least Squares Regression: mini-batching, averaging, and model misspecification
• Yes, but Did It Work?: Evaluating Variational Inference
• Monte Carlo Gradient Estimation in Machine Learning
• Consistency of Variational Bayes Inference for Estimation and Model Selection in Mixtures
• The Heavy-Tail Phenomenon in SGD
• On Convergence-Diagnostic based Step Sizes for Stochastic Gradient Descent
• Anytime Tail Averaging

Cited by in corpus (7)

• Variational Bayes survival analysis for unemployment modelling
• Pathfinder: Parallel quasi-Newton variational inference
• Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy
• Hamiltonian Monte Carlo using an adjoint-differentiated Laplace approximation: Bayesian inference for latent Gaussian models and beyond
• Challenges and Opportunities in High-dimensional Variational Inference
• Meta-analysis of dichotomous and ordinal tests without a gold standard
• Relay Variational Inference: A Method for Accelerated Encoderless VI