Weight Uncertainty in Neural Networks

Abstract

We introduce a new, efficient, principled and backpropagation-compatible algorithm for learning a probability distribution on the weights of a neural network, called Bayes by Backprop. It regularises the weights by minimising a compression cost, known as the variational free energy or the expected lower bound on the marginal likelihood. We show that this principled kind of regularisation yields comparable performance to dropout on MNIST classification. We then demonstrate how the learnt uncertainty in the weights can be used to improve generalisation in non-linear regression problems, and how this weight uncertainty can be used to drive the exploration-exploitation trade-off in reinforcement learning.

In Proceedings of the 32nd International Conference on Machine Learning (ICML 2015)

References in corpus (3)

• Distilling the Knowledge in a Neural Network
• Improving neural networks by preventing co-adaptation of feature detectors
• Further Optimal Regret Bounds for Thompson Sampling

Cited by in corpus (192)

• Deep and Confident Prediction for Time Series at Uber
• A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
• Peer-to-peer Federated Learning on Graphs
• Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Space
• A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks
• Quality of Uncertainty Quantification for Bayesian Neural Network Inference
• Non-Intrusive Reduced-Order Modeling Using Uncertainty-Aware Deep Neural Networks and Proper Orthogonal Decomposition: Application to Flood Modeling
• A Survey of Active Learning for Text Classification using Deep Neural Networks
• Reinforcement Learning through Active Inference
• Contrastive Training for Improved Out-of-Distribution Detection
• An Equivalence of Fully Connected Layer and Convolutional Layer
• Model Selection in Bayesian Neural Networks via Horseshoe Priors
• Transient-optimised real-bogus classification with Bayesian Convolutional Neural Networks -- sifting the GOTO candidate stream
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• 'In-Between' Uncertainty in Bayesian Neural Networks
• Noisy Softmax: Improving the Generalization Ability of DCNN via Postponing the Early Softmax Saturation
• Uncertainty-Aware Reliable Text Classification
• Uncertainty Decomposition in Bayesian Neural Networks with Latent Variables
• Improving Local Identifiability in Probabilistic Box Embeddings
• Learnable Bernoulli Dropout for Bayesian Deep Learning
• Subspace Inference for Bayesian Deep Learning
• Deep Robust Kalman Filter
• On Hallucination and Predictive Uncertainty in Conditional Language Generation
• Bayesian Semisupervised Learning with Deep Generative Models
• Physics-Constrained Bayesian Neural Network for Fluid Flow Reconstruction with Sparse and Noisy Data
• Trust Issues: Uncertainty Estimation Does Not Enable Reliable OOD Detection On Medical Tabular Data
• AutoOD: Automated Outlier Detection via Curiosity-guided Search and Self-imitation Learning
• Uncertainty quantification of molecular property prediction with Bayesian neural networks
• Bayesian Policy Gradients via Alpha Divergence Dropout Inference
• Bridging the Gap Between Explainable AI and Uncertainty Quantification to Enhance Trustability
• Efficient exploration with Double Uncertain Value Networks
• Deep Gaussian Processes: A Survey
• An Empirical Analysis of the Impact of Data Augmentation on Knowledge Distillation
• Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions
• Benchmarking the Neural Linear Model for Regression
• Variational Inference to Measure Model Uncertainty in Deep Neural Networks
• Estimating Model Uncertainty of Neural Networks in Sparse Information Form
• Distance-Based Learning from Errors for Confidence Calibration
• Bayesian Learning of Neural Network Architectures
• Calibration of Model Uncertainty for Dropout Variational Inference
• Contextual Dropout: An Efficient Sample-Dependent Dropout Module
• Towards calibrated and scalable uncertainty representations for neural networks
• Bayesian active learning for production, a systematic study and a reusable library
• Deep Bayesian Active Learning for Multiple Correct Outputs
• Meta-Learning surrogate models for sequential decision making
• Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networks
• Hypermodels for Exploration
• Bayesian Neural Networks
• Routing Networks with Co-training for Continual Learning
• Quantifying sources of uncertainty in drug discovery predictions with probabilistic models
• Kalman meets Bellman: Improving Policy Evaluation through Value Tracking
• Machine learning at the atomic-scale
• Bayesian Graph Neural Networks for Molecular Property Prediction
• One Versus all for deep Neural Network Incertitude (OVNNI) quantification
• Revisiting One-vs-All Classifiers for Predictive Uncertainty and Out-of-Distribution Detection in Neural Networks
• Individual Claims Forecasting with Bayesian Mixture Density Networks
• Bayesian Inference with Certifiable Adversarial Robustness
• Almost Surely Stable Deep Dynamics
• Prior choice affects ability of Bayesian neural networks to identify unknowns
• Radial and Directional Posteriors for Bayesian Neural Networks
• Tied Hidden Factors in Neural Networks for End-to-End Speaker Recognition
• Stochastic Maximum Likelihood Optimization via Hypernetworks
• Uncertainty-Aware Lookahead Factor Models for Quantitative Investing
• A Convergence Analysis for A Class of Practical Variance-Reduction Stochastic Gradient MCMC
• Mix and Match: A Novel FPGA-Centric Deep Neural Network Quantization Framework
• A Scale Mixture Perspective of Multiplicative Noise in Neural Networks
• Continual Learning in Deep Neural Network by Using a Kalman Optimiser
• Remote sensing image fusion based on Bayesian GAN
• Bayesian Neural Networks at Scale: A Performance Analysis and Pruning Study
• Bayesian x-vector: Bayesian Neural Network based x-vector System for Speaker Verification
• Consistent Sparse Deep Learning: Theory and Computation
• Simultaneously Reconciled Quantile Forecasting of Hierarchically Related Time Series
• Conditional Variational Autoencoder for Learned Image Reconstruction
• Fast learning rate of deep learning via a kernel perspective
• Comparing Semi-Parametric Model Learning Algorithms for Dynamic Model Estimation in Robotics
• Semi-Supervised Hypothesis Transfer for Source-Free Domain Adaptation
• On architectural choices in deep learning: From network structure to gradient convergence and parameter estimation
• On Stein Variational Neural Network Ensembles
• Meta Learning Black-Box Population-Based Optimizers
• Task Embedding Temporal Convolution Networks for Transfer Learning Problems in Renewable Power Time-Series Forecast
• Trust Region Value Optimization using Kalman Filtering
• $M{C}^{2}RAM$: Markov Chain Monte Carlo Sampling in SRAM for Fast Bayesian Inference
• Bayesian Autoencoders: Analysing and Fixing the Bernoulli likelihood for Out-of-Distribution Detection
• Ensemble Model Patching: A Parameter-Efficient Variational Bayesian Neural Network
• Bayesian Structure Adaptation for Continual Learning
• The Benefit of the Doubt: Uncertainty Aware Sensing for Edge Computing Platforms
• JAXNS: a high-performance nested sampling package based on JAX
• On Batch Normalisation for Approximate Bayesian Inference
• Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights
• A benchmark study on reliable molecular supervised learning via Bayesian learning
• Quantum Bayesian Neural Networks
• Applying SVGD to Bayesian Neural Networks for Cyclical Time-Series Prediction and Inference
• Energy Confused Adversarial Metric Learning for Zero-Shot Image Retrieval and Clustering
• Efficient Ensemble Model Generation for Uncertainty Estimation with Bayesian Approximation in Segmentation
• Multi-Loss Sub-Ensembles for Accurate Classification with Uncertainty Estimation
• Probabilistically-autoencoded horseshoe-disentangled multidomain item-response theory models
• Beyond Point Estimate: Inferring Ensemble Prediction Variation from Neuron Activation Strength in Recommender Systems
• Combining Model and Parameter Uncertainty in Bayesian Neural Networks
• Bayesian Neural Networks at Finite Temperature
• A Survey of Techniques All Classifiers Can Learn from Deep Networks: Models, Optimizations, and Regularization
• Efficient Computation Reduction in Bayesian Neural Networks Through Feature Decomposition and Memorization
• Adaptive Temporal-Difference Learning for Policy Evaluation with Per-State Uncertainty Estimates
• A Physics-Data-Driven Bayesian Method for Heat Conduction Problems
• Uncertainty-Aware Deep Classifiers using Generative Models
• Adaptively Preconditioned Stochastic Gradient Langevin Dynamics
• Gradient-Free Adversarial Attacks for Bayesian Neural Networks
• Low Complexity Approximate Bayesian Logistic Regression for Sparse Online Learning
• Estimation with Uncertainty via Conditional Generative Adversarial Networks
• A Generative Model for Sampling High-Performance and Diverse Weights for Neural Networks
• Characterizing Membership Privacy in Stochastic Gradient Langevin Dynamics
• Sampling-Free Learning of Bayesian Quantized Neural Networks
• Radial Prediction Layer
• Disentangling the Gauss-Newton Method and Approximate Inference for Neural Networks
• On Modelling Label Uncertainty in Deep Neural Networks: Automatic Estimation of Intra-observer Variability in 2D Echocardiography Quality Assessment
• Estimating Predictive Uncertainty Under Program Data Distribution Shift
• Probabilistic neural network-based reduced-order surrogate for fluid flows
• SkiNet: A Deep Learning Solution for Skin Lesion Diagnosis with Uncertainty Estimation and Explainability
• On the Calibration and Uncertainty of Neural Learning to Rank Models
• Bayesian Recurrent Framework for Missing Data Imputation and Prediction with Clinical Time Series
• Neural Likelihoods for Multi-Output Gaussian Processes
• A Novel Regression Loss for Non-Parametric Uncertainty Optimization
• Probabilistic Discriminative Learning with Layered Graphical Models
• Approaching Neural Network Uncertainty Realism
• Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning
• Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect
• Safety and Robustness in Decision Making: Deep Bayesian Recurrent Neural Networks for Somatic Variant Calling in Cancer
• Bayesian Variational Optimization for Combinatorial Spaces
• Uncertainty-aware Remaining Useful Life predictor
• Uncertainty quantification of molecular property prediction using Bayesian neural network models
• Variational Bayes: A report on approaches and applications
• NP-ODE: Neural Process Aided Ordinary Differential Equations for Uncertainty Quantification of Finite Element Analysis
• Stabilising priors for robust Bayesian deep learning
• Unifying Variational Inference and PAC-Bayes for Supervised Learning that Scales
• Uncertainty Quantification in Deep Residual Neural Networks
• Pixel Invisibility: Detecting Objects Invisible in Color Images
• Performance Measurement for Deep Bayesian Neural Network
• Practical calibration of the temperature parameter in Gibbs posteriors
• Stochastic Gradient MCMC with Stale Gradients
• Accurate and Reliable Forecasting using Stochastic Differential Equations
• Understanding Uncertainty in Bayesian Deep Learning
• Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models
• Revisiting hard thresholding for DNN pruning
• RCoNet: Deformable Mutual Information Maximization and High-order Uncertainty-aware Learning for Robust COVID-19 Detection
• TyXe: Pyro-based Bayesian neural nets for Pytorch
• Bayesian analysis of the prevalence bias: learning and predicting from imbalanced data
• Bayesian Convolutional Neural Networks for Seven Basic Facial Expression Classifications
• Learning a Discriminant Latent Space with Neural Discriminant Analysis
• On Graph Neural Network Ensembles for Large-Scale Molecular Property Prediction
• Probability Paths and the Structure of Predictions over Time
• Sampling Prediction-Matching Examples in Neural Networks: A Probabilistic Programming Approach
• Variational Bayes Neural Network: Posterior Consistency, Classification Accuracy and Computational Challenges
• Bayesian Distributional Policy Gradients
• Sequential Anomaly Detection using Inverse Reinforcement Learning
• Nearest-Neighbor Neural Networks for Geostatistics
• Amortized Variational Deep Q Network
• Towards On-Chip Bayesian Neuromorphic Learning
• Know Where To Drop Your Weights: Towards Faster Uncertainty Estimation
• Inherent Noise in Gradient Based Methods
• Estimating the Uncertainty of Neural Network Forecasts for Influenza Prevalence Using Web Search Activity
• Class Incremental Online Streaming Learning
• Probabilistic prediction of the heave motions of a semi-submersible by a deep learning problem model
• $\Delta$-UQ: Accurate Uncertainty Quantification via Anchor Marginalization
• Uncertainty based Class Activation Maps for Visual Question Answering
• Bayesian deep learning of affordances from RGB images
• MPC-guided Imitation Learning of Neural Network Policies for the Artificial Pancreas
• Approximate Bayesian Optimisation for Neural Networks
• Structured DropConnect for Uncertainty Inference in Image Classification
• Deep Bayesian Recurrent Neural Networks for Somatic Variant Calling in Cancer
• T-SVDNet: Exploring High-Order Prototypical Correlations for Multi-Source Domain Adaptation
• Fundamental Issues Regarding Uncertainties in Artificial Neural Networks
• Bayesian Sampling Bias Correction: Training with the Right Loss Function
• Uncertainty in Neural Relational Inference Trajectory Reconstruction
• Influence Diagram Bandits: Variational Thompson Sampling for Structured Bandit Problems
• Probabilistic Human Motion Prediction via A Bayesian Neural Network
• On the generalization of bayesian deep nets for multi-class classification
• Transformation Models for Flexible Posteriors in Variational Bayes
• A Novel Unsupervised Post-Processing Calibration Method for DNNS with Robustness to Domain Shift
• Logit-based Uncertainty Measure in Classification
• A Bayesian Approach to Invariant Deep Neural Networks
• Ladder Polynomial Neural Networks
• Adaptive Bayesian Reticulum
• REST: Performance Improvement of a Black Box Model via RL-based Spatial Transformation
• Multiple Angles of Arrival Estimation using Neural Networks
• Comparison of Graphcore IPUs and Nvidia GPUsfor cosmology applications
• Implicit Priors for Knowledge Sharing in Bayesian Neural Networks
• The Effect of Prior Lipschitz Continuity on the Adversarial Robustness of Bayesian Neural Networks
• Uncertainty Estimation in Deep Neural Networks for Point Cloud Segmentation in Factory Planning
• Variational Information Bottleneck Model for Accurate Indoor Position Recognition
• Robustness Against Outliers For Deep Neural Networks By Gradient Conjugate Priors
• A Kernel Framework to Quantify a Model's Local Predictive Uncertainty under Data Distributional Shifts
• Decision Theoretic Bootstrapping
• Trust but Verify: Assigning Prediction Credibility by Counterfactual Constrained Learning