On Smoothing and Inference for Topic Models

Abstract

Latent Dirichlet analysis, or topic modeling, is a flexible latent variable framework for modeling high-dimensional sparse count data. Various learning algorithms have been developed in recent years, including collapsed Gibbs sampling, variational inference, and maximum a posteriori estimation, and this variety motivates the need for careful empirical comparisons. In this paper, we highlight the close connections between these approaches. We find that the main differences are attributable to the amount of smoothing applied to the counts. When the hyperparameters are optimized, the differences in performance among the algorithms diminish significantly. The ability of these algorithms to achieve solutions of comparable accuracy gives us the freedom to select computationally efficient approaches. Using the insights gained from this comparative study, we show how accurate topic models can be learned in several seconds on text corpora with thousands of documents.

Appears in Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI2009)

References in corpus (1)

• Discrete Component Analysis

Cited by in corpus (14)

• Fast Variational Inference in the Conjugate Exponential Family
• Structured Stochastic Variational Inference
• The supervised hierarchical Dirichlet process
• Stochastic Variational Inference
• Rethinking Collapsed Variational Bayes Inference for LDA
• Augment-and-Conquer Negative Binomial Processes
• Using Variational Inference and MapReduce to Scale Topic Modeling
• Rebuilding Factorized Information Criterion: Asymptotically Accurate Marginal Likelihood
• Supervised Blockmodelling
• Collapsed Variational Bayes Inference of Infinite Relational Model
• Managing sparsity, time, and quality of inference in topic models
• A Scalable Asynchronous Distributed Algorithm for Topic Modeling
• Variational Inference in Nonconjugate Models
• Topic Modeling of Hierarchical Corpora