Simple and Effective VAE Training with Calibrated Decoders

Abstract

Variational autoencoders (VAEs) provide an effective and simple method for modeling complex distributions. However, training VAEs often requires considerable hyperparameter tuning to determine the optimal amount of information retained by the latent variable. We study the impact of calibrated decoders, which learn the uncertainty of the decoding distribution and can determine this amount of information automatically, on the VAE performance. While many methods for learning calibrated decoders have been proposed, many of the recent papers that employ VAEs rely on heuristic hyperparameters and ad-hoc modifications instead. We perform the first comprehensive comparative analysis of calibrated decoder and provide recommendations for simple and effective VAE training. Our analysis covers a range of image and video datasets and several single-image and sequential VAE models. We further propose a simple but novel modification to the commonly used Gaussian decoder, which computes the prediction variance analytically. We observe empirically that using heuristic modifications is not necessary with our method. Project website is at https://orybkin.github.io/sigma-vae/

arXiv commentInternational Conference on Machine Learning (ICML), 2021. Project website is at https://orybkin.github.io/sigma-vae/

References in corpus (30)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Auto-Encoding Variational Bayes
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• On Calibration of Modern Neural Networks
• Toward Multimodal Image-to-Image Translation
• DRAW: A Recurrent Neural Network For Image Generation
• InfoVAE: Information Maximizing Variational Autoencoders
• Learning Latent Dynamics for Planning from Pixels
• Stochastic Video Generation with a Learned Prior
• Stochastic Adversarial Video Prediction
• Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow
• Towards a Neural Statistician
• Dream to Control: Learning Behaviors by Latent Imagination
• BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
• Jukebox: A Generative Model for Music
• Towards Conceptual Compression
• From Variational to Deterministic Autoencoders
• Diagnosing and Enhancing VAE Models
• Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Distribution Matching in Variational Inference
• Discrete Variational Autoencoders
• Fixing a Broken ELBO
• Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models
• Latent Space Oddity: on the Curvature of Deep Generative Models
• Temporal Difference Variational Auto-Encoder
• Variational Variance: Simple, Reliable, Calibrated Heteroscedastic Noise Variance Parameterization
• Rate-Regularization and Generalization in VAEs

Cited by in corpus (5)

• Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
• Re-parameterizing VAEs for stability
• High-dimensional Asymptotics of VAEs: Threshold of Posterior Collapse and Dataset-Size Dependence of Rate-Distortion Curve
• Imitative Planning using Conditional Normalizing Flow
• It's LeVAsa not LevioSA! Latent Encodings for Valence-Arousal Structure Alignment