Factor Analysis, Probabilistic Principal Component Analysis, Variational Inference, and Variational Autoencoder: Tutorial and Survey

Abstract

This is a tutorial and survey paper on factor analysis, probabilistic Principal Component Analysis (PCA), variational inference, and Variational Autoencoder (VAE). These methods, which are tightly related, are dimensionality reduction and generative models. They assume that every data point is generated from or caused by a low-dimensional latent factor. By learning the parameters of distribution of latent space, the corresponding low-dimensional factors are found for the sake of dimensionality reduction. For their stochastic and generative behaviour, these models can also be used for generation of new data points in the data space. In this paper, we first start with variational inference where we derive the Evidence Lower Bound (ELBO) and Expectation Maximization (EM) for learning the parameters. Then, we introduce factor analysis, derive its joint and marginal distributions, and work out its EM steps. Probabilistic PCA is then explained, as a special case of factor analysis, and its closed-form solutions are derived. Finally, VAE is explained where the encoder, decoder and sampling from the latent space are introduced. Training VAE using both EM and backpropagation are explained.

To appear as a part of an upcoming textbook on dimensionality reduction and manifold learning. v2: corrected some mathematical typos

References in corpus (9)

• Variational Autoencoder for Deep Learning of Images, Labels and Captions
• Towards Deeper Understanding of Variational Autoencoding Models
• Tutorial: Deriving the Standard Variational Autoencoder (VAE) Loss Function
• Eigenvalue and Generalized Eigenvalue Problems: Tutorial
• Unsupervised and Supervised Principal Component Analysis: Tutorial
• Sampling Algorithms, from Survey Sampling to Monte Carlo Methods: Tutorial and Literature Review
• Fitting A Mixture Distribution to Data: Tutorial
• Nearly Optimal Variational Inference for High Dimensional Regression with Shrinkage Priors
• Stochastic Neighbor Embedding with Gaussian and Student-t Distributions: Tutorial and Survey

Cited by in corpus (4)

• Theoretical Connection between Locally Linear Embedding, Factor Analysis, and Probabilistic PCA
• Generative Adversarial Networks and Adversarial Autoencoders: Tutorial and Survey
• Generative Locally Linear Embedding
• Sufficient Dimension Reduction for High-Dimensional Regression and Low-Dimensional Embedding: Tutorial and Survey