Tutorial: Deriving the Standard Variational Autoencoder (VAE) Loss Function

Abstract

In Bayesian machine learning, the posterior distribution is typically computationally intractable, hence variational inference is often required. In this approach, an evidence lower bound on the log likelihood of data is maximized during training. Variational Autoencoders (VAE) are one important example where variational inference is utilized. In this tutorial, we derive the variational lower bound loss function of the standard variational autoencoder. We do so in the instance of a gaussian latent prior and gaussian approximate posterior, under which assumptions the Kullback-Leibler term in the variational lower bound has a closed form solution. We derive essentially everything we use along the way; everything from Bayes' theorem to the Kullback-Leibler divergence.

References in corpus (3)

• Towards Principled Methods for Training Generative Adversarial Networks
• Symmetric Variational Autoencoder and Connections to Adversarial Learning
• retina-VAE: Variationally Decoding the Spectrum of Macular Disease

Cited by in corpus (6)

• Accelerating Bayesian microseismic event location with deep learning
• Deciphering the Scattering of Mechanically Driven Polymers using Deep Learning
• Factor Analysis, Probabilistic Principal Component Analysis, Variational Inference, and Variational Autoencoder: Tutorial and Survey
• Exploiting Latent Codes: Interactive Fashion Product Generation, Similar Image Retrieval, and Cross-Category Recommendation using Variational Autoencoders
• Maximum Entropy Auto-Encoding
• Deep generative demixing: Recovering Lipschitz signals from noisy subgaussian mixtures