WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling

Abstract

To train an inference network jointly with a deep generative topic model, making it both scalable to big corpora and fast in out-of-sample prediction, we develop Weibull hybrid autoencoding inference (WHAI) for deep latent Dirichlet allocation, which infers posterior samples via a hybrid of stochastic-gradient MCMC and autoencoding variational Bayes. The generative network of WHAI has a hierarchy of gamma distributions, while the inference network of WHAI is a Weibull upward-downward variational autoencoder, which integrates a deterministic-upward deep neural network, and a stochastic-downward deep generative model based on a hierarchy of Weibull distributions. The Weibull distribution can be used to well approximate a gamma distribution with an analytic Kullback-Leibler divergence, and has a simple reparameterization via the uniform noise, which help efficiently compute the gradients of the evidence lower bound with respect to the parameters of the inference network. The effectiveness and efficiency of WHAI are illustrated with experiments on big corpora.

arXiv commentICLR 2018

References in corpus (9)

• Nested Hierarchical Dirichlet Processes
• Neural Variational Inference and Learning in Belief Networks
• Beta-Negative Binomial Process and Poisson Factor Analysis
• Autoencoding Variational Inference For Topic Models
• Variational Bayesian Inference with Stochastic Search
• PixelVAE: A Latent Variable Model for Natural Images
• Stochastic gradient variational Bayes for gamma approximating distributions
• Deep Latent Dirichlet Allocation with Topic-Layer-Adaptive Stochastic Gradient Riemannian MCMC
• The Poisson Gamma Belief Network

Cited by in corpus (14)

• Variational Temporal Deep Generative Model for Radar HRRP Target Recognition
• The Dynamic Embedded Topic Model
• Dirichlet belief networks for topic structure learning
• A Discrete Variational Recurrent Topic Model without the Reparametrization Trick
• Dirichlet Graph Variational Autoencoder
• vONTSS: vMF based semi-supervised neural topic modeling with optimal transport
• Matching Visual Features to Hierarchical Semantic Topics for Image Paragraph Captioning
• Topic Modelling Meets Deep Neural Networks: A Survey
• Neural Topic Model via Optimal Transport
• Bayesian Structure Adaptation for Continual Learning
• Deep Poisson gamma dynamical systems
• Naive Gabor Networks for Hyperspectral Image Classification
• Bayesian Attention Modules
• Variational Hetero-Encoder Randomized GANs for Joint Image-Text Modeling