Deep Autoencoding Topic Model with Scalable Hybrid Bayesian Inference

Abstract

To build a flexible and interpretable model for document analysis, we develop deep autoencoding topic model (DATM) that uses a hierarchy of gamma distributions to construct its multi-stochastic-layer generative network. In order to provide scalable posterior inference for the parameters of the generative network, we develop topic-layer-adaptive stochastic gradient Riemannian MCMC that jointly learns simplex-constrained global parameters across all layers and topics, with topic and layer specific learning rates. Given a posterior sample of the global parameters, in order to efficiently infer the local latent representations of a document under DATM across all stochastic layers, we propose a Weibull upward-downward variational encoder that deterministically propagates information upward via a deep neural network, followed by a Weibull distribution based stochastic downward generative model. To jointly model documents and their associated labels, we further propose supervised DATM that enhances the discriminative power of its latent representations. The efficacy and scalability of our models are demonstrated on both unsupervised and supervised learning tasks on big corpora.

arXiv commentTo appear in IEEE Transactions on Pattern Analysis and Machine Intelligence. arXiv admin note: text overlap with arXiv:1803.01328

References in corpus (12)

• Supervised Topic Models
• Nested Hierarchical Dirichlet Processes
• Neural Variational Inference and Learning in Belief Networks
• Beta-Negative Binomial Process and Poisson Factor Analysis
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• Autoencoding Variational Inference For Topic Models
• Variational Bayesian Inference with Stochastic Search
• Gibbs Max-margin Topic Models with Data Augmentation
• Stochastic gradient variational Bayes for gamma approximating distributions
• Max-margin Deep Generative Models
• The Poisson Gamma Belief Network
• Parsimonious Bayesian deep networks