Iterative Neural Autoregressive Distribution Estimator (NADE-k)

Abstract

Training of the neural autoregressive density estimator (NADE) can be viewed as doing one step of probabilistic inference on missing values in data. We propose a new model that extends this inference scheme to multiple steps, arguing that it is easier to learn to improve a reconstruction in $k$ steps rather than to learn to reconstruct in a single inference step. The proposed model is an unsupervised building block for deep learning that combines the desirable properties of NADE and multi-predictive training: (1) Its test likelihood can be computed analytically, (2) it is easy to generate independent samples from it, and (3) it uses an inference engine that is a superset of variational inference for Boltzmann machines. The proposed NADE-k is competitive with the state-of-the-art in density estimation on the two datasets tested.

Accepted at Neural Information Processing Systems (NIPS) 2014

References in corpus (5)

• ADADELTA: An Adaptive Learning Rate Method
• Theano: new features and speed improvements
• Better Mixing via Deep Representations
• Deep AutoRegressive Networks
• A Deep and Tractable Density Estimator

Cited by in corpus (12)

• Pixel Recurrent Neural Networks
• Professor Forcing: A New Algorithm for Training Recurrent Networks
• Ladder Variational Autoencoders
• Z-Forcing: Training Stochastic Recurrent Networks
• Neural Autoregressive Distribution Estimation
• The Variational Gaussian Process
• LogitBoost autoregressive networks
• Supervised Anomaly Detection based on Deep Autoregressive Density Estimators
• Learning Deep Generative Models with Doubly Stochastic MCMC
• Locally Masked Convolution for Autoregressive Models
• Projected Latent Markov Chain Monte Carlo: Conditional Sampling of Normalizing Flows
• Semi-Implicit Stochastic Recurrent Neural Networks