Variational Variance: Simple, Reliable, Calibrated Heteroscedastic Noise Variance Parameterization

Abstract

Brittle optimization has been observed to adversely impact model likelihoods for regression and VAEs when simultaneously fitting neural network mappings from a (random) variable onto the mean and variance of a dependent Gaussian variable. Previous works have bolstered optimization and improved likelihoods, but fail other basic posterior predictive checks (PPCs). Under the PPC framework, we propose critiques to test predictive mean and variance calibration and the predictive distribution's ability to generate sensible data. We find that our attractively simple solution, to treat heteroscedastic variance variationally, sufficiently regularizes variance to pass these PPCs. We consider a diverse gamut of existing and novel priors and find our methods preserve or outperform existing model likelihoods while significantly improving parameter calibration and sample quality for regression and VAEs.

arXiv comment17 pages, 6 figures, 10 tables

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Neural Discrete Representation Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Implicit Reparameterization Gradients
• Functional Variational Bayesian Neural Networks
• VAE with a VampPrior
• Reliable training and estimation of variance networks

Cited by in corpus (4)

• Simple and Effective VAE Training with Calibrated Decoders
• Integrating Uncertainty into Neural Network-based Speech Enhancement
• Addressing Variance Shrinkage in Variational Autoencoders using Quantile Regression
• Deep Classifiers with Label Noise Modeling and Distance Awareness