Backpropagation for Implicit Spectral Densities
arXiv:1806.00499
Abstract
Most successful machine intelligence systems rely on gradient-based learning, which is made possible by backpropagation. Some systems are designed to aid us in interpreting data when explicit goals cannot be provided. These unsupervised systems are commonly trained by backpropagating through a likelihood function. We introduce a tool that allows us to do this even when the likelihood is not explicitly set, by instead using the implicit likelihood of the model. Explicitly defining the likelihood often entails making heavy-handed assumptions that impede our ability to solve challenging tasks. On the other hand, the implicit likelihood of the model is accessible without the need for such assumptions. Our tool, which we call spectral backpropagation, allows us to optimize it in much greater generality than what has been attempted before. GANs can also be viewed as a technique for optimizing implicit likelihoods. We study them using spectral backpropagation in order to demonstrate robustness for high-dimensional problems, and identify two novel properties of the generator G: (1) there exist aberrant, nonsensical outputs to which G assigns very high likelihood, and (2) the eigenvectors of the metric induced by G over latent space correspond to quasi-disentangled explanatory factors.
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Intriguing properties of neural networks
- WaveNet: A Generative Model for Raw Audio
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Conditional Image Generation with PixelCNN Decoders
- Wide Residual Networks
- Pixel Recurrent Neural Networks
- Improved Techniques for Training GANs
- Density estimation using Real NVP
- MADE: Masked Autoencoder for Distribution Estimation
- Sylvester Normalizing Flows for Variational Inference
- Improving Variational Auto-Encoders using Householder Flow
- On the Effects of Batch and Weight Normalization in Generative Adversarial Networks
- Gradient Estimators for Implicit Models