Deep Directed Generative Autoencoders

Abstract

For discrete data, the likelihood $P(x)$ can be rewritten exactly and parametrized into $P(X=x)=P(X=x|H=f(x))P(H=f(x))$ if $P(X|H)$ has enough capacity to put no probability mass on any $x^{\prime }$ for which $f(x^{\prime })\ne f(x)$, where $f(\cdot )$ is a deterministic discrete function. The log of the first factor gives rise to the log-likelihood reconstruction error of an autoencoder with $f(\cdot )$ as the encoder and $P(X|H)$ as the (probabilistic) decoder. The log of the second term can be seen as a regularizer on the encoded activations $h=f(x)$, e.g., as in sparse autoencoders. Both encoder and decoder can be represented by a deep neural network and trained to maximize the average of the optimal log-likelihood $\log p(x)$. The objective is to learn an encoder $f(\cdot )$ that maps $X$ to $f(X)$ that has a much simpler distribution than $X$ itself, estimated by $P(H)$. This "flattens the manifold" or concentrates probability mass in a smaller number of (relevant) dimensions over which the distribution factorizes. Generating samples from the model is straightforward using ancestral sampling. One challenge is that regular back-propagation cannot be used to obtain the gradient on the parameters of the encoder, but we find that using the straight-through estimator works well here. We also find that although optimizing a single level of such architecture may be difficult, much better results can be obtained by pre-training and stacking them, gradually transforming the data distribution into one that is more easily captured by a simple parametric model.

References in corpus (10)

• Auto-Encoding Variational Bayes
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Theano: new features and speed improvements
• Better Mixing via Deep Representations
• How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
• Deep AutoRegressive Networks
• A Deep and Tractable Density Estimator
• Estimating or Propagating Gradients Through Stochastic Neurons
• Reweighted Wake-Sleep

Cited by in corpus (6)

• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• NICE: Non-linear Independent Components Estimation
• Generating Out of Distribution Adversarial Attack using Latent Space Poisoning
• Learning to learn generative programs with Memoised Wake-Sleep
• GSNs : Generative Stochastic Networks
• Note on Equivalence Between Recurrent Neural Network Time Series Models and Variational Bayesian Models