A correlated topic model of Science

Abstract

Topic models, such as latent Dirichlet allocation (LDA), can be useful tools for the statistical analysis of document collections and other discrete data. The LDA model assumes that the words of each document arise from a mixture of topics, each of which is a distribution over the vocabulary. A limitation of LDA is the inability to model topic correlation even though, for example, a document about genetics is more likely to also be about disease than X-ray astronomy. This limitation stems from the use of the Dirichlet distribution to model the variability among the topic proportions. In this paper we develop the correlated topic model (CTM), where the topic proportions exhibit correlation via the logistic normal distribution [J. Roy. Statist. Soc. Ser. B 44 (1982) 139--177]. We derive a fast variational inference algorithm for approximate posterior inference in this model, which is complicated by the fact that the logistic normal is not conjugate to the multinomial. We apply the CTM to the articles from Science published from 1990--1999, a data set that comprises 57M words. The CTM gives a better fit of the data than LDA, and we demonstrate its use as an exploratory tool of large document collections.

Published at http://dx.doi.org/10.1214/07-AOAS114 in the Annals of Applied Statistics (http://www.imstat.org/aoas/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (3)

• High-dimensional graphs and variable selection with the Lasso
• Describing disability through individual-level mixture models for multivariate binary data
• A Generalized Mean Field Algorithm for Variational Inference in Exponential Families

Cited by in corpus (23)

• Variational Inference: A Review for Statisticians
• Variational inference for large-scale models of discrete choice
• Exploration and Exploitation of Victorian Science in Darwin's Reading Notebooks
• Topic Discovery via Latent Space Clustering of Pretrained Language Model Representations
• Proper motions in the VVV Survey: Results for more than 15 million stars across NGC 6544
• Mixed-Membership of Experts Stochastic Blockmodel
• Granularity of algorithmically constructed publication-level classifications of research publications: Identification of topics
• Semantic and Relational Spaces in Science of Science: Deep Learning Models for Article Vectorisation
• Model trees with topic model preprocessing: An approach for data journalism illustrated with the WikiLeaks Afghanistan war logs
• Exponential Family Mixed Membership Models for Soft~Clustering of Multivariate Data
• Semiparametric Regression using Variational Approximations
• Partial-Mastery Cognitive Diagnosis Models
• Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details
• Algorithmic labeling in hierarchical classifications of publications: Evaluation of bibliographic fields and term weighting approaches
• Mining DEV for social and technical insights about software development
• Efficient Clustering from Distributions over Topics
• Hierarchical Dirichlet Scaling Process
• Scalable Cross-lingual Document Similarity through Language-specific Concept Hierarchies
• Higher Criticism for Discriminating Word-Frequency Tables and Testing Authorship
• Information retrieval and structural complexity of legal trees
• How research programs come apart: the example of supersymmetry and the disunity of physics
• Generalized Decomposition Priors on R2
• bioSBM: a random graph model to integrate epigenomic data in chromatin structure prediction