Inferring Parameters and Structure of Latent Variable Models by Variational Bayes

Abstract

Current methods for learning graphical models with latent variables and a fixed structure estimate optimal values for the model parameters. Whereas this approach usually produces overfitting and suboptimal generalization performance, carrying out the Bayesian program of computing the full posterior distributions over the parameters remains a difficult problem. Moreover, learning the structure of models with latent variables, for which the Bayesian approach is crucial, is yet a harder problem. In this paper I present the Variational Bayes framework, which provides a solution to these problems. This approach approximates full posterior distributions over model parameters and structures, as well as latent variables, in an analytical manner without resorting to sampling methods. Unlike in the Laplace approximation, these posteriors are generally non-Gaussian and no Hessian needs to be computed. The resulting algorithm generalizes the standard Expectation Maximization algorithm, and its convergence is guaranteed. I demonstrate that this algorithm can be applied to a large class of models in several domains, including unsupervised clustering and blind source separation.

Appears in Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence (UAI1999)

References in corpus (1)

• The Bayesian Structural EM Algorithm

Cited by in corpus (43)

• An introduction to sampling via measure transport
• Bayesian Evidence and Model Selection
• Complex sequencing rules of birdsong can be explained by simple hidden Markov processes
• A Survey on Latent Tree Models and Applications
• Distributed Variational Bayesian Algorithms Over Sensor Networks
• Generic decoding of seen and imagined objects using hierarchical visual features
• Variational Inference for Generalized Linear Mixed Models Using Partially Noncentered Parametrizations
• Stochastic Variational Inference
• Posterior Mean Super-resolution with a Causal Gaussian Markov Random Field Prior
• Receiver Architectures for MIMO-OFDM Based on a Combined VMP-SP Algorithm
• Stochastic variational inference for large-scale discrete choice models using adaptive batch sizes
• Variational inference for sparse spectrum Gaussian process regression
• Variational approximation for mixtures of linear mixed models
• Gaussian variational approximation for high-dimensional state space models
• The variational Laplace approach to approximate Bayesian inference
• Asymptotic Accuracy of Distribution-Based Estimation for Latent Variables
• A stochastic variational framework for fitting and diagnosing generalized linear mixed models
• Deterministic Approximate EM Algorithm; Application to the Riemann Approximation EM and the Tempered EM
• Quantum Annealing for Variational Bayes Inference
• Fast Inference for Intractable Likelihood Problems using Variational Bayes
• Convergence and asymptotic normality of variational Bayesian approximations for exponential family models with missing values
• Rebuilding Factorized Information Criterion: Asymptotically Accurate Marginal Likelihood
• Variational inference based on a subclass of closed skew normals
• Variational Inference for Sparse Gaussian Process Modulated Hawkes Process
• The Inverse G-Wishart Distribution and Variational Message Passing
• Online reinforcement learning with sparse rewards through an active inference capsule
• Efficient variational inference for generalized linear mixed models with large datasets
• Posterior Mean Super-Resolution with a Compound Gaussian Markov Random Field Prior
• Maximum likelihood estimation of a finite mixture of logistic regression models in a continuous data stream
• Nonparametric Reduced-Rank Regression for Multi-SNP, Multi-Trait Association Mapping
• Variational Bayesian Inference For A Scale Mixture Of Normal Distributions Handling Missing Data
• Approximating Incomplete Kernel Matrices by the em Algorithm
• Gaussian variational approximation with a factor covariance structure
• A Factorial Mixture Prior for Compositional Deep Generative Models
• Empirical Bayes Linked Matrix Decomposition
• Accuracy of Latent-Variable Estimation in Bayesian Semi-Supervised Learning
• Effects of Additional Data on Bayesian Clustering
• Grammar Induction for Minimalist Grammars using Variational Bayesian Inference : A Technical Report
• Multi-Decoder RNN Autoencoder Based on Variational Bayes Method
• Hidden Markov Models Applied To Intraday Momentum Trading With Side Information
• Incorporating prior information and borrowing information in high-dimensional sparse regression using the horseshoe and variational Bayes
• Semisupervised hyperspectral image unmixing using a variational Bayes algorithm
• Variational Bayesian Inference and Complexity Control for Stochastic Block Models