Consistency of Variational Bayes Inference for Estimation and Model Selection in Mixtures

Abstract

Mixture models are widely used in Bayesian statistics and machine learning, in particular in computational biology, natural language processing and many other fields. Variational inference, a technique for approximating intractable posteriors thanks to optimization algorithms, is extremely popular in practice when dealing with complex models such as mixtures. The contribution of this paper is two-fold. First, we study the concentration of variational approximations of posteriors, which is still an open problem for general mixtures, and we derive consistency and rates of convergence. We also tackle the problem of model selection for the number of components: we study the approach already used in practice, which consists in maximizing a numerical criterion (the Evidence Lower Bound). We prove that this strategy indeed leads to strong oracle inequalities. We illustrate our theoretical results by applications to Gaussian and multinomial mixtures.

References in corpus (14)

• Variational Inference: A Review for Statisticians
• Rényi Divergence and Kullback-Leibler Divergence
• Model-based Clustering
• Frequentist Consistency of Variational Bayes
• Inference and Evaluation of the Multinomial Mixture Model for Text Clustering
• On the properties of variational approximations of Gibbs posteriors
• Consistency of Variational Bayes Inference for Estimation and Model Selection in Mixtures
• 1-bit Matrix Completion: PAC-Bayesian Analysis of a Variational Approximation
• Tuning Tempered Transitions
• Optimal estimation of Gaussian mixtures via denoised method of moments
• Convergence Rates of Variational Posterior Distributions
• Bayesian inference of Gaussian mixture models with noninformative priors
• Bayesian fractional posteriors
• Simultaneous Dimension Reduction and Clustering via the NMF-EM Algorithm

Cited by in corpus (24)

• User-friendly introduction to PAC-Bayes bounds
• A Primer on PAC-Bayesian Learning
• Consistency of Variational Bayes Inference for Estimation and Model Selection in Mixtures
• MMD-Bayes: Robust Bayesian Estimation via Maximum Mean Discrepancy
• Finite sample properties of parametric MMD estimation: robustness to misspecification and dependence
• Practical bounds on the error of Bayesian posterior approximations: A nonasymptotic approach
• Robust, Accurate Stochastic Optimization for Variational Inference
• Learning Attribute Patterns in High-Dimensional Structured Latent Attribute Models
• Dynamics of coordinate ascent variational inference: A case study in 2D Ising models
• Consistency of ELBO maximization for model selection
• Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details
• Convergence Rates of Variational Inference in Sparse Deep Learning
• A Generalization Bound for Online Variational Inference
• Bayesian Model Selection via Mean-Field Variational Approximation
• Convergence Rates of Variational Posterior Distributions
• Variational Inference in high-dimensional linear regression
• Validated Variational Inference via Practical Posterior Error Bounds
• Adaptive variational Bayes: Optimality, computation and applications
• Frequentist Consistency of Generalized Variational Inference
• Sparse Variational Inference: Bayesian Coresets from Scratch
• Non-exponentially weighted aggregation: regret bounds for unbounded loss functions
• Mixture weights optimisation for Alpha-Divergence Variational Inference
• On the Statistical Consistency of Risk-Sensitive Bayesian Decision-Making
• Variational excess risk bound for general state space models