Manifold Optimization Assisted Gaussian Variational Approximation

Abstract

Gaussian variational approximation is a popular methodology to approximate posterior distributions in Bayesian inference especially in high dimensional and large data settings. To control the computational cost while being able to capture the correlations among the variables, the low rank plus diagonal structure was introduced in the previous literature for the Gaussian covariance matrix. For a specific Bayesian learning task, the uniqueness of the solution is usually ensured by imposing stringent constraints on the parameterized covariance matrix, which could break down during the optimization process. In this paper, we consider two special covariance structures by applying the Stiefel manifold and Grassmann manifold constraints, to address the optimization difficulty in such factorization architectures. To speed up the updating process with minimum hyperparameter-tuning efforts, we design two new schemes of Riemannian stochastic gradient descent methods and compare them with other existing methods of optimizing on manifolds. In addition to fixing the identification issue, results from both simulation and empirical experiments prove the ability of the proposed methods of obtaining competitive accuracy and comparable converge speed in both high-dimensional and large-scale learning tasks.

arXiv comment31 pages

References in corpus (16)

• Auto-Encoding Variational Bayes
• ADADELTA: An Adaptive Learning Rate Method
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Manopt, a Matlab toolbox for optimization on manifolds
• Linear Dimensionality Reduction: Survey, Insights, and Generalizations
• Clustering on Multi-Layer Graphs via Subspace Analysis on Grassmann Manifolds
• Low-rank optimization for semidefinite convex problems
• Riemannian Adaptive Optimization Methods
• Conic geometric optimisation on the manifold of positive definite matrices
• Variational Bayesian Inference with Stochastic Search
• Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds
• Boosting Variational Inference
• Nonparametric variational inference
• Grassmannian Learning: Embedding Geometry Awareness in Shallow and Deep Learning
• SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient
• Riemannian adaptive stochastic gradient algorithms on matrix manifolds