Bridging the ELBO and MMD

Abstract

One of the challenges in training generative models such as the variational auto encoder (VAE) is avoiding posterior collapse. When the generator has too much capacity, it is prone to ignoring latent code. This problem is exacerbated when the dataset is small, and the latent dimension is high. The root of the problem is the ELBO objective, specifically the Kullback-Leibler (KL) divergence term in objective function \citep{zhao2019infovae}. This paper proposes a new objective function to replace the KL term with one that emulates the maximum mean discrepancy (MMD) objective. It also introduces a new technique, named latent clipping, that is used to control distance between samples in latent space. A probabilistic autoencoder model, named $\mu$-VAE, is designed and trained on MNIST and MNIST Fashion datasets, using the new objective function and is shown to outperform models trained with ELBO and $\beta$-VAE objective. The $\mu$-VAE is less prone to posterior collapse, and can generate reconstructions and new samples in good quality. Latent representations learned by $\mu$-VAE are shown to be good and can be used for downstream tasks such as classification.

arXiv comment14 pages, 11 figures

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Neural Discrete Representation Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Autoencoding beyond pixels using a learned similarity metric
• Understanding disentangling in $\beta$-VAE
• IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis
• Towards Deeper Understanding of Variational Autoencoding Models
• VAE with a VampPrior
• Avoiding Latent Variable Collapse With Generative Skip Models
• Semi-Amortized Variational Autoencoders
• Preventing Posterior Collapse with delta-VAEs

Cited by in corpus (1)

• Inverse Graphics: Unsupervised Learning of 3D Shapes from Single Images