Variance Networks: When Expectation Does Not Meet Your Expectations

Abstract

Ordinary stochastic neural networks mostly rely on the expected values of their weights to make predictions, whereas the induced noise is mostly used to capture the uncertainty, prevent overfitting and slightly boost the performance through test-time averaging. In this paper, we introduce variance layers, a different kind of stochastic layers. Each weight of a variance layer follows a zero-mean distribution and is only parameterized by its variance. We show that such layers can learn surprisingly well, can serve as an efficient exploration tool in reinforcement learning tasks and provide a decent defense against adversarial attacks. We also show that a number of conventional Bayesian neural networks naturally converge to such zero-mean posteriors. We observe that in these cases such zero-mean parameterization leads to a much better training objective than conventional parameterizations where the mean is being learned.

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Weight Uncertainty in Neural Networks
• Noisy Networks for Exploration
• Stochastic Gradient Hamiltonian Monte Carlo
• Variational Dropout and the Local Reparameterization Trick
• Parameter Space Noise for Exploration
• Predictive Uncertainty Estimation via Prior Networks
• Structured Bayesian Pruning via Log-Normal Multiplicative Noise
• Variational Gaussian Dropout is not Bayesian

Cited by in corpus (5)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Averaging Weights Leads to Wider Optima and Better Generalization
• A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
• A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks
• Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference