On Smoothing and Inference for Topic Models

Abstract

Latent Dirichlet analysis, or topic modeling, is a flexible latent variable framework for modeling high-dimensional sparse count data. Various learning algorithms have been developed in recent years, including collapsed Gibbs sampling, variational inference, and maximum a posteriori estimation, and this variety motivates the need for careful empirical comparisons. In this paper, we highlight the close connections between these approaches. We find that the main differences are attributable to the amount of smoothing applied to the counts. When the hyperparameters are optimized, the differences in performance among the algorithms diminish significantly. The ability of these algorithms to achieve solutions of comparable accuracy gives us the freedom to select computationally efficient approaches. Using the insights gained from this comparative study, we show how accurate topic models can be learned in several seconds on text corpora with thousands of documents.

Appears in Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI2009)

References in corpus (1)

• Discrete Component Analysis

Cited by in corpus (53)

• What is Wrong with Topic Modeling? (and How to Fix it Using Search-based Software Engineering)
• Beta-Negative Binomial Process and Poisson Factor Analysis
• Streaming Variational Bayes
• On Estimation and Selection for Topic Models
• Representation Learning for Natural Language Processing
• Fast Variational Inference in the Conjugate Exponential Family
• Learning Topic Models by Belief Propagation
• Structured Stochastic Variational Inference
• Finding Trends in Software Research
• The supervised hierarchical Dirichlet process
• A Topic Modeling Toolbox Using Belief Propagation
• Bayesian Nonparametric Multilevel Clustering with Group-Level Contexts
• Stochastic Variational Inference
• WarpLDA: a Cache Efficient O(1) Algorithm for Latent Dirichlet Allocation
• Fast Online EM for Big Topic Modeling
• End-to-end Learning of LDA by Mirror-Descent Back Propagation over a Deep Architecture
• Clustering by Low-Rank Doubly Stochastic Matrix Decomposition
• Computing Web-scale Topic Models using an Asynchronous Parameter Server
• Rethinking Collapsed Variational Bayes Inference for LDA
• Augment-and-Conquer Negative Binomial Processes
• A New Approach to Speeding Up Topic Modeling
• Using Variational Inference and MapReduce to Scale Topic Modeling
• A Coefficient of Determination for Probabilistic Topic Models
• Unsupervised Terminological Ontology Learning based on Hierarchical Topic Modeling
• Rebuilding Factorized Information Criterion: Asymptotically Accurate Marginal Likelihood
• A General Method for Robust Bayesian Modeling
• Towards Big Topic Modeling
• Guaranteed inference in topic models
• Supervised Blockmodelling
• Image Retrieval with a Bayesian Model of Relevance Feedback
• Rookie: A unique approach for exploring news archives
• Collapsed Variational Bayes Inference of Infinite Relational Model
• Residual Belief Propagation for Topic Modeling
• Managing sparsity, time, and quality of inference in topic models
• Inference in topic models: sparsity and trade-off
• Supervised multi-specialist topic model with applications on large-scale electronic health record data
• Quantifying Latent Moral Foundations in Twitter Narratives: The Case of the Syrian White Helmets Misinformation
• An MCMC Approach to Empirical Bayes Inference and Bayesian Sensitivity Analysis via Empirical Processes
• Blocking Collapsed Gibbs Sampler for Latent Dirichlet Allocation Models
• Stochastic Collapsed Variational Bayesian Inference for Latent Dirichlet Allocation
• A Scalable Asynchronous Distributed Algorithm for Topic Modeling
• Variational Inference in Nonconjugate Models
• The Discrete Infinite Logistic Normal Distribution
• Statistical inference on errorfully observed graphs
• ATM:Adversarial-neural Topic Model
• Bilingual Topic Models for Comparable Corpora
• Stochastic Divergence Minimization for Biterm Topic Model
• Hierarchical learning of grids of microtopics
• ALBU: An approximate Loopy Belief message passing algorithm for LDA to improve performance on small data sets
• Stochastic Collapsed Variational Inference for Sequential Data
• TopicModel4J: A Java Package for Topic Models
• Prior-aware Dual Decomposition: Document-specific Topic Inference for Spectral Topic Models
• Topic Modeling of Hierarchical Corpora