Inference Suboptimality in Variational Autoencoders

Abstract

Amortized inference allows latent-variable models trained via variational learning to scale to large datasets. The quality of approximate inference is determined by two factors: a) the capacity of the variational distribution to match the true posterior and b) the ability of the recognition network to produce good variational parameters for each datapoint. We examine approximate inference in variational autoencoders in terms of these factors. We find that divergence from the true posterior is often due to imperfect recognition networks, rather than the limited complexity of the approximating distribution. We show that this is due partly to the generator learning to accommodate the choice of approximation. Furthermore, we show that the parameters used to increase the expressiveness of the approximation play a role in generalizing inference rather than simply improving the complexity of the approximation.

ICML

References in corpus (3)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
• The Reversible Residual Network: Backpropagation Without Storing Activations

Cited by in corpus (39)

• Data Augmentation in High Dimensional Low Sample Size Setting Using a Geometry-Based Variational Autoencoder
• Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis
• A Deep Learning Algorithm for High-Dimensional Exploratory Item Factor Analysis
• Semi-Amortized Variational Autoencoders
• A Tutorial on Deep Latent Variable Models of Natural Language
• Gaussian Process Conditional Density Estimation
• Amortized Variational Inference: A Systematic Review
• Likelihood-free MCMC with Amortized Approximate Ratio Estimators
• COIN: COmpression with Implicit Neural representations
• Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse
• Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection
• Scalable Gaussian Process Variational Autoencoders
• Variational Laplace Autoencoders
• Reweighted Expectation Maximization
• Bayesian Learning of Neural Network Architectures
• Adaptive Path-Integral Autoencoder: Representation Learning and Planning for Dynamical Systems
• Leveraging the Exact Likelihood of Deep Latent Variable Models
• Regularization-Agnostic Compressed Sensing MRI Reconstruction with Hypernetworks
• Balancing Reconstruction Quality and Regularisation in ELBO for VAEs
• Hierarchical VAEs Know What They Don't Know
• Independent Subspace Analysis for Unsupervised Learning of Disentangled Representations
• Unsupervised Video Decomposition using Spatio-temporal Iterative Inference
• Geometry-Aware Hamiltonian Variational Auto-Encoder
• Implicit Deep Latent Variable Models for Text Generation
• Failure Modes of Variational Autoencoders and Their Effects on Downstream Tasks
• A Batch Normalized Inference Network Keeps the KL Vanishing Away
• Control as Hybrid Inference
• Iterative VAE as a predictive brain model for out-of-distribution generalization
• Pseudo-Encoded Stochastic Variational Inference
• Variationally Inferred Sampling Through a Refined Bound for Probabilistic Programs
• Multi-Source Neural Variational Inference
• BooVAE: Boosting Approach for Continual Learning of VAE
• Unsupervised Representation Learning via Neural Activation Coding
• Meta Cyclical Annealing Schedule: A Simple Approach to Avoiding Meta-Amortization Error
• Counterfactual Explanations via Latent Space Projection and Interpolation
• Tomographic Auto-Encoder: Unsupervised Bayesian Recovery of Corrupted Data
• Relay Variational Inference: A Method for Accelerated Encoderless VI
• Mutual Information Constraints for Monte-Carlo Objectives
• Contributions to Large Scale Bayesian Inference and Adversarial Machine Learning