Autoencoding Variational Inference For Topic Models

Abstract

Topic models are one of the most popular methods for learning representations of text, but a major challenge is that any change to the topic model requires mathematically deriving a new inference algorithm. A promising approach to address this problem is autoencoding variational Bayes (AEVB), but it has proven diffi- cult to apply to topic models in practice. We present what is to our knowledge the first effective AEVB based inference method for latent Dirichlet allocation (LDA), which we call Autoencoded Variational Inference For Topic Model (AVITM). This model tackles the problems caused for AEVB by the Dirichlet prior and by component collapsing. We find that AVITM matches traditional methods in accuracy with much better inference time. Indeed, because of the inference network, we find that it is unnecessary to pay the computational cost of running variational optimization on test data. Because AVITM is black box, it is readily applied to new topic models. As a dramatic illustration of this, we present a new topic model called ProdLDA, that replaces the mixture model in LDA with a product of experts. By changing only one line of code from LDA, we find that ProdLDA yields much more interpretable topics, even if LDA is trained via collapsed Gibbs sampling.

Cited by in corpus (33)

• Open Domain Event Extraction Using Neural Latent Variable Models
• WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling
• Classification Aware Neural Topic Model and its Application on a New COVID-19 Disinformation Corpus
• Advances in Variational Inference
• News-Driven Stock Prediction With Attention-Based Noisy Recurrent State Transition
• Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence
• Coupled VAE: Improved Accuracy and Robustness of a Variational Autoencoder
• A Discrete Variational Recurrent Topic Model without the Reparametrization Trick
• Learning document embeddings along with their uncertainties
• Dirichlet Graph Variational Autoencoder
• Topic-Driven and Knowledge-Aware Transformer for Dialogue Emotion Detection
• Matching Visual Features to Hierarchical Semantic Topics for Image Paragraph Captioning
• Emerging App Issue Identification via Online Joint Sentiment-Topic Tracing
• ETC-NLG: End-to-end Topic-Conditioned Natural Language Generation
• DrNAS: Dirichlet Neural Architecture Search
• Neural Topic Model via Optimal Transport
• Leveraging Just a Few Keywords for Fine-Grained Aspect Detection Through Weakly Supervised Co-Training
• Prediction-Constrained Training for Semi-Supervised Mixture and Topic Models
• Auto-Encoding Variational Bayes for Inferring Topics and Visualization
• Deep Autoencoding Topic Model with Scalable Hybrid Bayesian Inference
• Text Modeling with Syntax-Aware Variational Autoencoders
• Dynamic Topic Language Model on Heterogeneous Children's Mental Health Clinical Notes
• Topic-Guided Abstractive Multi-Document Summarization
• A New Distribution on the Simplex with Auto-Encoding Applications
• ATM:Adversarial-neural Topic Model
• Discovering topics with neural topic models built from PLSA assumptions
• Towards Verified Stochastic Variational Inference for Probabilistic Programs
• What You Say and How You Say it: Joint Modeling of Topics and Discourse in Microblog Conversations
• BioNLP-OST 2019 RDoC Tasks: Multi-grain Neural Relevance Ranking Using Topics and Attention Based Query-Document-Sentence Interactions
• Multi-view and Multi-source Transfers in Neural Topic Modeling with Pretrained Topic and Word Embeddings
• Adversarial Learning of Poisson Factorisation Model for Gauging Brand Sentiment in User Reviews
• Learning Interpretable and Discrete Representations with Adversarial Training for Unsupervised Text Classification
• Neural Attention-Aware Hierarchical Topic Model