Noise Contrastive Variational Autoencoders

Abstract

We take steps towards understanding the "posterior collapse (PC)" difficulty in variational autoencoders (VAEs),~i.e. a degenerate optimum in which the latent codes become independent of their corresponding inputs. We rely on calculus of variations and theoretically explore a few popular VAE models, showing that PC always occurs for non-parametric encoders and decoders. Inspired by the popular noise contrastive estimation algorithm, we propose NC-VAE where the encoder discriminates between the latent codes of real data and of some artificially generated noise, in addition to encouraging good data reconstruction abilities. Theoretically, we prove that our model cannot reach PC and provide novel lower bounds. Our method is straightforward to implement and has the same run-time as vanilla VAE. Empirically, we showcase its benefits on popular image and text datasets.

There is a mistake common to all the main proofs. In summary, what we find are saddle points or global maxima of the respective loss functions and not the global minima. We apologize for this

References in corpus (15)

• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Conditional Image Generation with PixelCNN Decoders
• Pixel Recurrent Neural Networks
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• InfoVAE: Information Maximizing Variational Autoencoders
• Improving Variational Inference with Inverse Autoregressive Flow
• Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
• Towards Deeper Understanding of Variational Autoencoding Models
• Contrastive Variational Autoencoder Enhances Salient Features
• Fixing a Broken ELBO
• Avoiding Latent Variable Collapse With Generative Skip Models
• MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders
• Preventing Posterior Collapse with delta-VAEs
• Bounded Information Rate Variational Autoencoders
• Variational Noise-Contrastive Estimation