Analytic natural gradient updates for Cholesky factor in Gaussian variational approximation

Abstract

Natural gradients can improve convergence in stochastic variational inference significantly but inverting the Fisher information matrix is daunting in high dimensions. Moreover, in Gaussian variational approximation, natural gradient updates of the precision matrix do not ensure positive definiteness. To tackle this issue, we derive analytic natural gradient updates of the Cholesky factor of the covariance or precision matrix, and consider sparsity constraints representing different posterior correlation structures. Stochastic normalized natural gradient ascent with momentum is proposed for implementation in generalized linear mixed models and deep neural networks.

References in corpus (7)

• New insights and perspectives on the natural gradient method
• Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression
• Variational Bayes with Synthetic Likelihood
• Gaussian variational approximation with sparse precision matrices
• Variational Inference for Generalized Linear Mixed Models Using Partially Noncentered Parametrizations
• A Simple Convergence Proof of Adam and Adagrad
• Are you using test log-likelihood correctly?

Cited by in corpus (2)

• Variational inference based on a subclass of closed skew normals
• Non-conjugate variational Bayes for pseudo-likelihood mixed effect models