The Dynamic Embedded Topic Model

Abstract

Topic modeling analyzes documents to learn meaningful patterns of words. For documents collected in sequence, dynamic topic models capture how these patterns vary over time. We develop the dynamic embedded topic model (D-ETM), a generative model of documents that combines dynamic latent Dirichlet allocation (D-LDA) and word embeddings. The D-ETM models each word with a categorical distribution parameterized by the inner product between the word embedding and a per-time-step embedding representation of its assigned topic. The D-ETM learns smooth topic trajectories by defining a random walk prior over the embedding representations of the topics. We fit the D-ETM using structured amortized variational inference with a recurrent neural network. On three different corpora---a collection of United Nations debates, a set of ACL abstracts, and a dataset of Science Magazine articles---we found that the D-ETM outperforms D-LDA on a document completion task. We further found that the D-ETM learns more diverse and coherent topics than D-LDA while requiring significantly less time to fit.

References in corpus (12)

• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Auto-Encoding Variational Bayes
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Supervised Topic Models
• Continuous Time Dynamic Topic Models
• Dynamic Word Embeddings
• Jointly Learning Word Embeddings and Latent Topics
• WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling
• Deep Latent Dirichlet Allocation with Topic-Layer-Adaptive Stochastic Gradient Riemannian MCMC
• Distilled Wasserstein Learning for Word Embedding and Topic Modeling
• Scalable Generalized Dynamic Topic Models

Cited by in corpus (10)

• Improving Document Classification with Multi-Sense Embeddings
• Topic Space Trajectories: A case study on machine learning literature
• GRADE: Graph Dynamic Embedding
• Gaussian Hierarchical Latent Dirichlet Allocation: Bringing Polysemy Back
• VSEC-LDA: Boosting Topic Modeling with Embedded Vocabulary Selection
• P-SIF: Document Embeddings Using Partition Averaging
• Measuring diachronic sense change: new models and Monte Carlo methods for Bayesian inference
• Sequence-to-Sequence Learning with Latent Neural Grammars
• Jointly Dynamic Topic Model for Recognition of Lead-lag Relationship in Two Text Corpora
• Automatic Monitoring Social Dynamics During Big Incidences: A Case Study of COVID-19 in Bangladesh