Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks

Abstract

Large multilayer neural networks trained with backpropagation have recently achieved state-of-the-art results in a wide range of problems. However, using backprop for neural net learning still has some disadvantages, e.g., having to tune a large number of hyperparameters to the data, lack of calibrated probabilistic predictions, and a tendency to overfit the training data. In principle, the Bayesian approach to learning neural networks does not have these problems. However, existing Bayesian techniques lack scalability to large dataset and network sizes. In this work we present a novel scalable method for learning Bayesian neural networks, called probabilistic backpropagation (PBP). Similar to classical backpropagation, PBP works by computing a forward propagation of probabilities through the network and then doing a backward computation of gradients. A series of experiments on ten real-world datasets show that PBP is significantly faster than other techniques, while offering competitive predictive abilities. Our experiments also show that PBP provides accurate estimates of the posterior variance on the network weights.

References in corpus (4)

• Sequence to Sequence Learning with Neural Networks
• Practical Bayesian Optimization of Machine Learning Algorithms
• Deep Image: Scaling up Image Recognition
• Expectation Propagation for Neural Networks with Sparsity-promoting Priors

Cited by in corpus (192)

• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification
• Deep Exploration via Bootstrapped DQN
• Variational Dropout and the Local Reparameterization Trick
• Deep and Confident Prediction for Time Series at Uber
• Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm
• A General Framework for Uncertainty Estimation in Deep Learning
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Bayesian Dark Knowledge
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach
• Fault Detection and Identification using Bayesian Recurrent Neural Networks
• Bayesian Recurrent Neural Networks
• Deep Learning: A Bayesian Perspective
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• What Are Bayesian Neural Network Posteriors Really Like?
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• Balanced Sparsity for Efficient DNN Inference on GPU
• Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Space
• A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks
• Quality of Uncertainty Quantification for Bayesian Neural Network Inference
• Non-Intrusive Reduced-Order Modeling Using Uncertainty-Aware Deep Neural Networks and Proper Orthogonal Decomposition: Application to Flood Modeling
• Task Agnostic Continual Learning Using Online Variational Bayes
• Bayesian Batch Active Learning as Sparse Subset Approximation
• Implicit Weight Uncertainty in Neural Networks
• How to Evaluate Uncertainty Estimates in Machine Learning for Regression?
• Generalized Variational Inference: Three arguments for deriving new Posteriors
• Resource-efficient Deep Neural Networks for Automotive Radar Interference Mitigation
• A Perspective on Deep Learning for Molecular Modeling and Simulations
• Ensemble Neural Networks (ENN): A gradient-free stochastic method
• Bayesian Physics-Informed Neural Networks for the Subsurface Tomography based on the Eikonal Equation
• Supervised and Semi-supervised Deep Probabilistic Models for Indoor Positioning Problems
• Differentiable Compositional Kernel Learning for Gaussian Processes
• On the Selection of Initialization and Activation Function for Deep Neural Networks
• On the Expressiveness of Approximate Inference in Bayesian Neural Networks
• Getting a CLUE: A Method for Explaining Uncertainty Estimates
• Finite Versus Infinite Neural Networks: an Empirical Study
• Uncertainty in Neural Networks: Approximately Bayesian Ensembling
• Conditional Density Estimation with Bayesian Normalising Flows
• Natural-Parameter Networks: A Class of Probabilistic Neural Networks
• Can You Trust This Prediction? Auditing Pointwise Reliability After Learning
• The Functional Neural Process
• Qualitative Analysis of Monte Carlo Dropout
• 'In-Between' Uncertainty in Bayesian Neural Networks
• Information Aware Max-Norm Dirichlet Networks for Predictive Uncertainty Estimation
• Robust Simulation-Based Inference in Cosmology with Bayesian Neural Networks
• Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
• Client Selection for Federated Bayesian Learning
• Uncertainty-guided Continual Learning with Bayesian Neural Networks
• On the Importance of Strong Baselines in Bayesian Deep Learning
• ProbAct: A Probabilistic Activation Function for Deep Neural Networks
• Bayesian Neural Network Priors Revisited
• Uncertainty Decomposition in Bayesian Neural Networks with Latent Variables
• HyperGAN: A Generative Model for Diverse, Performant Neural Networks
• The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
• Reliable Neural Networks for Regression Uncertainty Estimation
• On the role of synaptic stochasticity in training low-precision neural networks
• Jensen-Shannon Divergence Based Novel Loss Functions for Bayesian Neural Networks
• Bayesian Inference with Anchored Ensembles of Neural Networks, and Application to Exploration in Reinforcement Learning
• DC-Check: A Data-Centric AI checklist to guide the development of reliable machine learning systems
• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation
• Sampling-Free Variational Inference of Bayesian Neural Networks by Variance Backpropagation
• Task Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updates
• A Survey on Bayesian Deep Learning
• Bayesian Semisupervised Learning with Deep Generative Models
• Bayesian Hypernetworks
• Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Quantifying Uncertainties in Natural Language Processing Tasks
• PBODL : Parallel Bayesian Online Deep Learning for Click-Through Rate Prediction in Tencent Advertising System
• Reliable training and estimation of variance networks
• Bayesian Graph Convolutional Neural Networks using Node Copying
• Deep learning via message passing algorithms based on belief propagation
• Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions
• Treatment-aware Diffusion Probabilistic Model for Longitudinal MRI Generation and Diffuse Glioma Growth Prediction
• Benchmarking the Neural Linear Model for Regression
• Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details
• A general framework for ensemble distribution distillation
• Estimating Model Uncertainty of Neural Networks in Sparse Information Form
• PIVEN: A Deep Neural Network for Prediction Intervals with Specific Value Prediction
• Bayesian Deep Learning via Subnetwork Inference
• Deep Evidential Regression
• Predictive Uncertainty Quantification with Compound Density Networks
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning
• Exploring Bayesian Deep Learning for Urgent Instructor Intervention Need in MOOC Forums
• Automatic structured variational inference
• Nonlinear Statistical Learning with Truncated Gaussian Graphical Models
• Bayesian Graph Convolutional Neural Networks Using Non-Parametric Graph Learning
• A Financial Service Chatbot based on Deep Bidirectional Transformers
• Anomalous Example Detection in Deep Learning: A Survey
• Radial and Directional Posteriors for Bayesian Neural Networks
• Classification Confidence Estimation with Test-Time Data-Augmentation
• Variational Variance: Simple, Reliable, Calibrated Heteroscedastic Noise Variance Parameterization
• Robust Deep Gaussian Processes
• Data Augmentation for Bayesian Deep Learning
• A Contour Stochastic Gradient Langevin Dynamics Algorithm for Simulations of Multi-modal Distributions
• Digital Twin-Assisted Robust and Adaptive Resource Slicing in LEO Satellite Networks
• Exploring Uncertainty in Deep Learning for Construction of Prediction Intervals
• Stochastic Neural Network with Kronecker Flow
• Non-Parametric Graph Learning for Bayesian Graph Neural Networks
• Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling
• Gaussian Gated Linear Networks
• Evaluating Scalable Uncertainty Estimation Methods for DNN-Based Molecular Property Prediction
• Variational Refinement for Importance Sampling Using the Forward Kullback-Leibler Divergence
• A Locally Adaptive Interpretable Regression
• Mitigating Uncertainty in Document Classification
• Ensemble Model Patching: A Parameter-Efficient Variational Bayesian Neural Network
• RoNGBa: A Robustly Optimized Natural Gradient Boosting Training Approach with Leaf Number Clipping
• NOMU: Neural Optimization-based Model Uncertainty
• Tractable Approximate Gaussian Inference for Bayesian Neural Networks
• Walsh-Hadamard Variational Inference for Bayesian Deep Learning
• Efficient Ensemble Model Generation for Uncertainty Estimation with Bayesian Approximation in Segmentation
• Quantum Bayesian Neural Networks
• Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights
• Generalized Bayesian Posterior Expectation Distillation for Deep Neural Networks
• Applying SVGD to Bayesian Neural Networks for Cyclical Time-Series Prediction and Inference
• AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantification
• Scalable Training of Inference Networks for Gaussian-Process Models
• Principled Pruning of Bayesian Neural Networks through Variational Free Energy Minimization
• On Batch Normalisation for Approximate Bayesian Inference
• TRADI: Tracking deep neural network weight distributions for uncertainty estimation
• Sampling-Free Learning of Bayesian Quantized Neural Networks
• Uncertainty-Aware Model Adaptation for Unsupervised Cross-Domain Object Detection
• Learnable Uncertainty under Laplace Approximations
• Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit
• Efficient Evaluation-Time Uncertainty Estimation by Improved Distillation
• Fast Adaptive Weight Noise
• Estimation of Accurate and Calibrated Uncertainties in Deterministic models
• The Monte Carlo Transformer: a stochastic self-attention model for sequence prediction
• Few-shot Conformal Prediction with Auxiliary Tasks
• Towards a Kernel based Uncertainty Decomposition Framework for Data and Models
• Adaptive Bayesian Linear Regression for Automated Machine Learning
• Why have a Unified Predictive Uncertainty? Disentangling it using Deep Split Ensembles
• Efficient Online Bayesian Inference for Neural Bandits
• From Predictions to Decisions: Using Lookahead Regularization
• Prediction intervals for Deep Neural Networks
• The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration
• Prior Activation Distribution (PAD): A Versatile Representation to Utilize DNN Hidden Units
• Neural Likelihoods for Multi-Output Gaussian Processes
• High-Order Stochastic Gradient Thermostats for Bayesian Learning of Deep Models
• Variational Depth Search in ResNets
• Fully Bayesian Recurrent Neural Networks for Safe Reinforcement Learning
• Uncertainty-aware Remaining Useful Life predictor
• Provably-Robust Runtime Monitoring of Neuron Activation Patterns
• Streaming Probabilistic Deep Tensor Factorization
• Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures
• Mitigating the Effects of Non-Identifiability on Inference for Bayesian Neural Networks with Latent Variables
• Estimating Predictive Uncertainty Under Program Data Distribution Shift
• Structured Dropout Variational Inference for Bayesian Neural Networks
• Prediction-based Resource Allocation using Bayesian Neural Networks and Minimum Cost and Maximum Flow Algorithm
• Assessing Deep Neural Networks as Probability Estimators
• A Survey on Assessing the Generalization Envelope of Deep Neural Networks: Predictive Uncertainty, Out-of-distribution and Adversarial Samples
• CRAUM-Net: Contextual Recursive Attention with Uncertainty Modeling for Salient Object Detection
• The Bayesian Method of Tensor Networks
• Adaptive Variational Bayesian Inference for Sparse Deep Neural Network
• Efficient and Robust LiDAR-Based End-to-End Navigation
• TyXe: Pyro-based Bayesian neural nets for Pytorch
• AutoCP: Automated Pipelines for Accurate Prediction Intervals
• Accuracy-Reliability Cost Function for Empirical Variance Estimation
• Uncertainty Quantification in Deep Residual Neural Networks
• Stochastic Variational Inference via Upper Bound
• Assessing the Robustness of Bayesian Dark Knowledge to Posterior Uncertainty
• Uncertainty Surrogates for Deep Learning
• A Bayesian Approach to Invariant Deep Neural Networks
• GP-ConvCNP: Better Generalization for Convolutional Conditional Neural Processes on Time Series Data
• Stochastic Bayesian Neural Networks
• Unifying Local and Global Change Detection in Dynamic Networks
• Why Calibration Error is Wrong Given Model Uncertainty: Using Posterior Predictive Checks with Deep Learning
• MOFA: Modular Factorial Design for Hyperparameter Optimization
• High-Dimensional Bayesian Optimization via Semi-Supervised Learning with Optimized Unlabeled Data Sampling
• n-MeRCI: A new Metric to Evaluate the Correlation Between Predictive Uncertainty and True Error
• An Approximate Bayesian Long Short-Term Memory Algorithm for Outlier Detection
• Regularising Deep Networks with Deep Generative Models
• Uncertainty estimation under model misspecification in neural network regression
• Efficient Approximate Inference with Walsh-Hadamard Variational Inference
• Bayes-Adaptive Deep Model-Based Policy Optimisation
• Modeling continuous-time stochastic processes using $\mathcal{N}$-Curve mixtures
• An Adaptive Empirical Bayesian Method for Sparse Deep Learning
• Thompson Sampling via Local Uncertainty
• Kernel Identification Through Transformers
• Robustness Against Outliers For Deep Neural Networks By Gradient Conjugate Priors
• Interval Deep Learning for Uncertainty Quantification in Safety Applications
• Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite Networks
• Distribution Free Uncertainty for the Minimum Norm Solution of Over-parameterized Linear Regression
• Beyond Marginal Uncertainty: How Accurately can Bayesian Regression Models Estimate Posterior Predictive Correlations?
• Uncertainty-Aware Multi-Modal Ensembling for Severity Prediction of Alzheimer's Dementia
• Bayesian Meta-Learning Through Variational Gaussian Processes
• Improving Uncertainty Calibration via Prior Augmented Data
• Intrinsic uncertainties and where to find them