Rate-Regularization and Generalization in VAEs
arXiv:1911.04594
Abstract
Variational autoencoders optimize an objective that combines a reconstruction loss (the distortion) and a KL term (the rate). The rate is an upper bound on the mutual information, which is often interpreted as a regularizer that controls the degree of compression. We here examine whether inclusion of the rate also acts as an inductive bias that improves generalization. We perform rate-distortion analyses that control the strength of the rate term, the network capacity, and the difficulty of the generalization problem. Decreasing the strength of the rate paradoxically improves generalization in most settings, and reducing the mutual information typically leads to underfitting. Moreover, we show that generalization continues to improve even after the mutual information saturates, indicating that the gap on the bound (i.e. the KL divergence relative to the inference marginal) affects generalization. This suggests that the standard Gaussian prior is not an inductive bias that typically aids generalization, prompting work to understand what choices of priors improve generalization in VAEs.
References in corpus (15)
- Auto-Encoding Variational Bayes
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- The information bottleneck method
- Conditional Image Generation with PixelCNN Decoders
- Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
- Disentangling by Factorising
- Variational Lossy Autoencoder
- From optimal transport to generative modeling: the VEGAN cookbook
- PixelVAE: A Latent Variable Model for Natural Images
- Bias and Generalization in Deep Generative Models: An Empirical Study
- Amortized Inference Regularization
- Identity Crisis: Memorization and Generalization under Extreme Overparameterization
- Memorization in Overparameterized Autoencoders
- Evaluating Lossy Compression Rates of Deep Generative Models
- On Implicit Regularization in -VAEs