Stochastic gradient variational Bayes for gamma approximating distributions

Abstract

While stochastic variational inference is relatively well known for scaling inference in Bayesian probabilistic models, related methods also offer ways to circumnavigate the approximation of analytically intractable expectations. The key challenge in either setting is controlling the variance of gradient estimates: recent work has shown that for continuous latent variables, particularly multivariate Gaussians, this can be achieved by using the gradient of the log posterior. In this paper we apply the same idea to gamma distributed latent variables given gamma variational distributions, enabling straightforward "black box" variational inference in models where sparsity and non-negativity are appropriate. We demonstrate the method on a recently proposed gamma process model for network data, as well as a novel sparse factor analysis. We outperform generic sampling algorithms and the approach of using Gaussian variational distributions on transformed variables.

References in corpus (6)

• ADADELTA: An Adaptive Learning Rate Method
• Theano: new features and speed improvements
• Nonparametric Bayesian sparse factor models with application to gene expression modeling
• Variational Bayesian Inference with Stochastic Search
• Infinite Edge Partition Models for Overlapping Community Detection and Link Prediction
• Automatic Variational Inference in Stan

Cited by in corpus (20)

• Automatic Differentiation Variational Inference
• WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling
• Pathwise Derivatives Beyond the Reparameterization Trick
• Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models
• Amortized Variational Inference: A Systematic Review
• The Generalized Reparameterization Gradient
• Advances in Variational Inference
• DeepStability: A Study of Unstable Numerical Methods and Their Solutions in Deep Learning
• Reparameterization Gradients through Acceptance-Rejection Sampling Algorithms
• A Poisson Gamma Probabilistic Model for Latent Node-group Memberships in Dynamic Networks
• Handling the Positive-Definite Constraint in the Bayesian Learning Rule
• Variational Auto-encoder Based Bayesian Poisson Tensor Factorization for Sparse and Imbalanced Count Data
• Low Complexity Approximate Bayesian Logistic Regression for Sparse Online Learning
• Neuron ranking -- an informed way to condense convolutional neural networks architecture
• Scalable Bayesian dynamic covariance modeling with variational Wishart and inverse Wishart processes
• Deep Autoencoding Topic Model with Scalable Hybrid Bayesian Inference
• Sequential Matrix Completion
• Generalized Transformation-based Gradient
• Dirichlet Pruning for Neural Network Compression
• Viking: Variational Bayesian Variance Tracking