Deep AutoRegressive Networks

Abstract

We introduce a deep, generative autoencoder capable of learning hierarchies of distributed representations from data. Successive deep stochastic hidden layers are equipped with autoregressive connections, which enable the model to be sampled from quickly and exactly via ancestral sampling. We derive an efficient approximate parameter estimation method based on the minimum description length (MDL) principle, which can be seen as maximising a variational lower bound on the log-likelihood, with a feedforward neural network implementing approximate inference. We demonstrate state-of-the-art generative performance on a number of classic data sets: several UCI data sets, MNIST and Atari 2600 games.

Appears in Proceedings of the 31st International Conference on Machine Learning (ICML), Beijing, China, 2014

References in corpus (6)

• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Variational Bayesian Inference with Stochastic Search
• A Deep and Tractable Density Estimator
• Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields
• A Generative Process for Sampling Contractive Auto-Encoders
• Learning Representations by Maximizing Compression

Cited by in corpus (52)

• Auto-Encoding Variational Bayes
• An Introduction to Variational Autoencoders
• Conditional Image Generation with PixelCNN Decoders
• Pixel Recurrent Neural Networks
• NICE: Non-linear Independent Components Estimation
• Variational Inference with Normalizing Flows
• DRAW: A Recurrent Neural Network For Image Generation
• Semi-Supervised Learning with Ladder Networks
• MADE: Masked Autoencoder for Distribution Estimation
• Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images
• Neural Variational Inference and Learning in Belief Networks
• Gradient Estimation Using Stochastic Computation Graphs
• How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
• Generative Image Modeling Using Spatial LSTMs
• Unsupervised Learning of 3D Structure from Images
• One-Shot Generalization in Deep Generative Models
• Data Generation as Sequential Decision Making
• Stochastic Backpropagation through Mixture Density Distributions
• Semi-Latent GAN: Learning to generate and modify facial images from attributes
• MuProp: Unbiased Backpropagation for Stochastic Neural Networks
• Discretize-Optimize vs. Optimize-Discretize for Time-Series Regression and Continuous Normalizing Flows
• Reweighted Wake-Sleep
• Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning
• Max-margin Deep Generative Models
• Iterative Neural Autoregressive Distribution Estimator (NADE-k)
• Multi-Attribute Selectivity Estimation Using Deep Learning
• Neural Autoregressive Distribution Estimation
• Accurate and Conservative Estimates of MRF Log-likelihood using Reverse Annealing
• Transformation Autoregressive Networks
• Deep Directed Generative Autoencoders
• A Review of Learning with Deep Generative Models from Perspective of Graphical Modeling
• Estimating Gradients for Discrete Random Variables by Sampling without Replacement
• Learning to Generate with Memory
• The Variational Gaussian Process
• Reweighted Expectation Maximization
• Revisiting Spatial Invariance with Low-Rank Local Connectivity
• Learning Deep Generative Models with Doubly Stochastic MCMC
• LogitBoost autoregressive networks
• Locally Masked Convolution for Autoregressive Models
• Probabilistic Residual Learning for Aleatoric Uncertainty in Image Restoration
• NUIG-Shubhanker@Dravidian-CodeMix-FIRE2020: Sentiment Analysis of Code-Mixed Dravidian text using XLNet
• Intermediate Data Caching Optimization for Multi-Stage and Parallel Big Data Frameworks
• A Stochastic Decoder for Neural Machine Translation
• Max-Margin Deep Generative Models for (Semi-)Supervised Learning
• Deep Dynamic Factor Models
• Optimal Variance Control of the Score Function Gradient Estimator for Importance Weighted Bounds
• GSNs : Generative Stochastic Networks
• Channel-Recurrent Autoencoding for Image Modeling
• Kinetic samplers for neural quantum states
• Neuro-symbolic EDA-based Optimisation using ILP-enhanced DBNs
• Predictive Synthesis of Quantum Materials by Probabilistic Reinforcement Learning
• Learning Sparsity of Representations with Discrete Latent Variables