Cauchy-Schwarz Regularized Autoencoder

Abstract

Recent work in unsupervised learning has focused on efficient inference and learning in latent variables models. Training these models by maximizing the evidence (marginal likelihood) is typically intractable. Thus, a common approximation is to maximize the Evidence Lower BOund (ELBO) instead. Variational autoencoders (VAE) are a powerful and widely-used class of generative models that optimize the ELBO efficiently for large datasets. However, the VAE's default Gaussian choice for the prior imposes a strong constraint on its ability to represent the true posterior, thereby degrading overall performance. A Gaussian mixture model (GMM) would be a richer prior, but cannot be handled efficiently within the VAE framework because of the intractability of the Kullback-Leibler divergence for GMMs. We deviate from the common VAE framework in favor of one with an analytical solution for Gaussian mixture prior. To perform efficient inference for GMM priors, we introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs. This new objective allows us to incorporate richer, multi-modal priors into the autoencoding framework. We provide empirical studies on a range of datasets and show that our objective improves upon variational auto-encoding models in density estimation, unsupervised clustering, semi-supervised learning, and face analysis.

References in corpus (16)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Categorical Reparameterization with Gumbel-Softmax
• Neural Discrete Representation Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Semi-Supervised Learning with Deep Generative Models
• Variational Inference with Normalizing Flows
• Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
• Understanding disentangling in $\beta$-VAE
• Wasserstein Auto-Encoders
• Towards Deeper Understanding of Variational Autoencoding Models
• Sylvester Normalizing Flows for Variational Inference
• From Variational to Deterministic Autoencoders
• Improving Variational Auto-Encoders using Householder Flow
• On the Latent Space of Wasserstein Auto-Encoders
• A Prior of a Googol Gaussians: a Tensor Ring Induced Prior for Generative Models

Cited by in corpus (1)

• A Compositional Atlas of Tractable Circuit Operations: From Simple Transformations to Complex Information-Theoretic Queries