Fixing a Broken ELBO

Abstract

Recent work in unsupervised representation learning has focused on learning deep directed latent-variable models. Fitting these models by maximizing the marginal likelihood or evidence is typically intractable, thus a common approximation is to maximize the evidence lower bound (ELBO) instead. However, maximum likelihood training (whether exact or approximate) does not necessarily result in a good latent representation, as we demonstrate both theoretically and empirically. In particular, we derive variational lower and upper bounds on the mutual information between the input and the latent variable, and use these bounds to derive a rate-distortion curve that characterizes the tradeoff between compression and reconstruction accuracy. Using this framework, we demonstrate that there is a family of models with identical ELBO, but different quantitative and qualitative characteristics. Our framework also suggests a simple new method to ensure that latent variable models with powerful stochastic decoders do not ignore their latent code.

arXiv comment21 pages, 9 figures

References in corpus (20)

• Auto-Encoding Variational Bayes
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• The information bottleneck method
• Conditional Image Generation with PixelCNN Decoders
• End-to-end Optimized Image Compression
• Understanding deep learning requires rethinking generalization
• Opening the Black Box of Deep Neural Networks via Information
• Deep Variational Information Bottleneck
• MADE: Masked Autoencoder for Distribution Estimation
• Lossy Image Compression with Compressive Autoencoders
• Ladder Variational Autoencoders
• Importance Weighted Autoencoders
• Masked Autoregressive Flow for Density Estimation
• Improving Variational Inference with Inverse Autoregressive Flow
• Early Visual Concept Learning with Unsupervised Deep Learning
• Real-Time Adaptive Image Compression
• Rényi Divergence Variational Inference
• VAE with a VampPrior
• Improved Lossy Image Compression with Priming and Spatially Adaptive Bit Rates for Recurrent Networks
• Hierarchical Variational Models

Cited by in corpus (31)

• A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
• Video Compression With Rate-Distortion Autoencoders
• Isolating Sources of Disentanglement in Variational Autoencoders
• ODE${}^2$VAE: Deep generative second order ODEs with Bayesian neural networks
• Physics-informed deep generative models
• VAE with a VampPrior
• ControlVAE: Controllable Variational Autoencoder
• Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE
• Distribution Matching in Variational Inference
• Simple and Effective VAE Training with Calibrated Decoders
• The Usual Suspects? Reassessing Blame for VAE Posterior Collapse
• Information bottleneck through variational glasses
• Bayesian Neural Networks With Maximum Mean Discrepancy Regularization
• Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse
• Shaping Belief States with Generative Environment Models for RL
• PixelVAE++: Improved PixelVAE with Discrete Prior
• Behavior Priors for Efficient Reinforcement Learning
• Information Bottleneck and its Applications in Deep Learning
• Differentiable probabilistic models of scientific imaging with the Fourier slice theorem
• Bayesian Neural Networks at Scale: A Performance Analysis and Pruning Study
• Variational Auto-Encoder: not all failures are equal
• Learning Embeddings from Cancer Mutation Sets for Classification Tasks
• A Free-Energy Principle for Representation Learning
• Flatsomatic: A Method for Compression of Somatic Mutation Profiles in Cancer
• ControlVAE: Tuning, Analytical Properties, and Performance Analysis
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• Neural Communication Systems with Bandwidth-limited Channel
• Noise Contrastive Variational Autoencoders
• The Variational InfoMax Learning Objective
• Mutual Information Constraints for Monte-Carlo Objectives
• GILBO: One Metric to Measure Them All