Deep Latent Dirichlet Allocation with Topic-Layer-Adaptive Stochastic Gradient Riemannian MCMC

Abstract

It is challenging to develop stochastic gradient based scalable inference for deep discrete latent variable models (LVMs), due to the difficulties in not only computing the gradients, but also adapting the step sizes to different latent factors and hidden layers. For the Poisson gamma belief network (PGBN), a recently proposed deep discrete LVM, we derive an alternative representation that is referred to as deep latent Dirichlet allocation (DLDA). Exploiting data augmentation and marginalization techniques, we derive a block-diagonal Fisher information matrix and its inverse for the simplex-constrained global model parameters of DLDA. Exploiting that Fisher information matrix with stochastic gradient MCMC, we present topic-layer-adaptive stochastic gradient Riemannian (TLASGR) MCMC that jointly learns simplex-constrained global parameters across all layers and topics, with topic and layer specific learning rates. State-of-the-art results are demonstrated on big data sets.

Appearing in ICML 2017

References in corpus (9)

• Auto-Encoding Variational Bayes
• ADADELTA: An Adaptive Learning Rate Method
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Nested Hierarchical Dirichlet Processes
• A Complete Recipe for Stochastic Gradient MCMC
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• Revisiting Natural Gradient for Deep Networks
• Modeling Documents with Deep Boltzmann Machines
• Stochastic Quasi-Newton Langevin Monte Carlo

Cited by in corpus (9)

• WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling
• Variational Temporal Deep Generative Model for Radar HRRP Target Recognition
• ServeNet: A Deep Neural Network for Web Services Classification
• The Dynamic Embedded Topic Model
• A Survey on Bayesian Deep Learning
• Matching Visual Features to Hierarchical Semantic Topics for Image Paragraph Captioning
• Fast Simulation of Hyperplane-Truncated Multivariate Normal Distributions
• GO Gradient for Expectation-Based Objectives
• Variational Hetero-Encoder Randomized GANs for Joint Image-Text Modeling