Improving latent variable descriptiveness with AutoGen
arXiv:1806.04480
Abstract
Powerful generative models, particularly in Natural Language Modelling, are commonly trained by maximizing a variational lower bound on the data log likelihood. These models often suffer from poor use of their latent variable, with ad-hoc annealing factors used to encourage retention of information in the latent variable. We discuss an alternative and general approach to latent variable modelling, based on an objective that combines the data log likelihood as well as the likelihood of a perfect reconstruction through an autoencoder. Tying these together ensures by design that the latent variable captures information about the observations, whilst retaining the ability to generate well. Interestingly, though this approach is a priori unrelated to VAEs, the lower bound attained is identical to the standard VAE bound but with the addition of a simple pre-factor; thus, providing a formal interpretation of the commonly used, ad-hoc pre-factors in training VAEs.
arXiv comment8 pages, 2 figures, 5 tables
References in corpus (8)
- Auto-Encoding Variational Bayes
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Variational Autoencoder for Deep Learning of Images, Labels and Captions
- Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
- Variational Lossy Autoencoder
- TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency
- Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
- A Hybrid Convolutional Variational Autoencoder for Text Generation