The continuous Bernoulli: fixing a pervasive error in variational autoencoders

Abstract

Variational autoencoders (VAE) have quickly become a central tool in machine learning, applicable to a broad range of data types and latent variable models. By far the most common first step, taken by seminal papers and by core software libraries alike, is to model MNIST data using a deep network parameterizing a Bernoulli likelihood. This practice contains what appears to be and what is often set aside as a minor inconvenience: the pixel data is [0,1] valued, not {0,1} as supported by the Bernoulli likelihood. Here we show that, far from being a triviality or nuisance that is convenient to ignore, this error has profound importance to VAE, both qualitative and quantitative. We introduce and fully characterize a new [0,1]-supported, single parameter distribution: the continuous Bernoulli, which patches this pervasive bug in VAE. This distribution is not nitpicking; it produces meaningful performance improvements across a range of metrics and datasets, including sharper image samples, and suggests a broader class of performant VAE.

arXiv commentAccepted at NeurIPS 2019

References in corpus (10)

• Auto-Encoding Variational Bayes
• Categorical Reparameterization with Gumbel-Softmax
• Loopy Belief Propagation for Approximate Inference: An Empirical Study
• Tutorial on Variational Autoencoders
• Variational Inference with Normalizing Flows
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders
• PixelVAE: A Latent Variable Model for Natural Images
• Generalized Variational Inference: Three arguments for deriving new Posteriors
• Reducing Reparameterization Gradient Variance

Cited by in corpus (17)

• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• Generalized Variational Inference: Three arguments for deriving new Posteriors
• Deblending galaxies with Variational Autoencoders: a joint multi-band, multi-instrument approach
• Neural Feature Search for RGB-Infrared Person Re-Identification
• Robust Variational Autoencoder
• FretNet: Continuous-Valued Pitch Contour Streaming for Polyphonic Guitar Tablature Transcription
• The continuous categorical: a novel simplex-valued exponential family
• VAEs in the Presence of Missing Data
• Geometry-Aware Hamiltonian Variational Auto-Encoder
• Avoiding Side Effects in Complex Environments
• Certifiably Robust Variational Autoencoders
• Invertible Gaussian Reparameterization: Revisiting the Gumbel-Softmax
• Sparse Communication via Mixed Distributions
• On Neural Networks as Infinite Tree-Structured Probabilistic Graphical Models
• Amortised Learning by Wake-Sleep
• Probabilistic task modelling for meta-learning
• Auto-encoding GPS data to reveal individual and collective behaviour