Spherical Latent Spaces for Stable Variational Autoencoders

Abstract

A hallmark of variational autoencoders (VAEs) for text processing is their combination of powerful encoder-decoder models, such as LSTMs, with simple latent distributions, typically multivariate Gaussians. These models pose a difficult optimization problem: there is an especially bad local optimum where the variational posterior always equals the prior and the model does not use the latent variable at all, a kind of "collapse" which is encouraged by the KL divergence term of the objective. In this work, we experiment with another choice of latent distribution, namely the von Mises-Fisher (vMF) distribution, which places mass on the surface of the unit hypersphere. With this choice of prior and posterior, the KL divergence term now only depends on the variance of the vMF distribution, giving us the ability to treat it as a fixed hyperparameter. We show that doing so not only averts the KL collapse, but consistently gives better likelihoods than Gaussians across a range of modeling conditions, including recurrent language modeling and bag-of-words document modeling. An analysis of the properties of our vMF representations shows that they learn richer and more nuanced structures in their latent representations than their Gaussian counterparts.

arXiv commentTo appear in EMNLP 2018; 11 pages; Code release: https://github.com/jiacheng-xu/vmf_vae_nlp

References in corpus (9)

• WaveNet: A Generative Model for Raw Audio
• DRAW: A Recurrent Neural Network For Image Generation
• Style Transfer from Non-Parallel Text by Cross-Alignment
• Toward Controlled Generation of Text
• Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
• von Mises-Fisher Mixture Model-based Deep learning: Application to Face Verification
• A Hybrid Convolutional Variational Autoencoder for Text Generation
• Adversarially Regularized Autoencoders
• Variational Neural Machine Translation

Cited by in corpus (8)

• A Tutorial on Deep Latent Variable Models of Natural Language
• Variational Autoencoders with Riemannian Brownian Motion Priors
• Symmetric Spaces for Graph Embeddings: A Finsler-Riemannian Approach
• A Batch Normalized Inference Network Keeps the KL Vanishing Away
• Variational Diffusion Autoencoders with Random Walk Sampling
• Geodesics in fibered latent spaces: A geometric approach to learning correspondences between conditions
• A Bayesian Approach to Spherical Factor Analysis for Binary Data
• Embedding-reparameterization procedure for manifold-valued latent variables in generative models