Hidden Talents of the Variational Autoencoder

Abstract

Variational autoencoders (VAE) represent a popular, flexible form of deep generative model that can be stochastically fit to samples from a given random process using an information-theoretic variational bound on the true underlying distribution. Once so-obtained, the model can be putatively used to generate new samples from this distribution, or to provide a low-dimensional latent representation of existing samples. While quite effective in numerous application domains, certain important mechanisms which govern the behavior of the VAE are obfuscated by the intractable integrals and resulting stochastic approximations involved. Moreover, as a highly non-convex model, it remains unclear exactly how minima of the underlying energy relate to original design purposes. We attempt to better quantify these issues by analyzing a series of tractable special cases of increasing complexity. In doing so, we unveil interesting connections with more traditional dimensionality reduction models, as well as an intrinsic yet underappreciated propensity for robustly dismissing sparse outliers when estimating latent manifolds. With respect to the latter, we demonstrate that the VAE can be viewed as the natural evolution of recent robust PCA models, capable of learning nonlinear manifolds of unknown dimension obscured by gross corruptions.

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• The Augmented Lagrange Multiplier Method for Exact Recovery of Corrupted Low-Rank Matrices
• Pixel Recurrent Neural Networks
• Semi-Supervised Learning with Deep Generative Models
• The Loss Surfaces of Multilayer Networks
• Autoencoding beyond pixels using a learned similarity metric
• Auxiliary Deep Generative Models
• Ladder Variational Autoencoders
• Non-Convex Rank Minimization via an Empirical Bayesian Approach

Cited by in corpus (5)

• A learning-based method for solving ill-posed nonlinear inverse problems: a simulation study of Lung EIT
• The Usual Suspects? Reassessing Blame for VAE Posterior Collapse
• Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse
• Fork or Fail: Cycle-Consistent Training with Many-to-One Mappings
• There and Back Again: Unraveling the Variational Auto-Encoder