Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
arXiv:1502.05336
Abstract
Large multilayer neural networks trained with backpropagation have recently achieved state-of-the-art results in a wide range of problems. However, using backprop for neural net learning still has some disadvantages, e.g., having to tune a large number of hyperparameters to the data, lack of calibrated probabilistic predictions, and a tendency to overfit the training data. In principle, the Bayesian approach to learning neural networks does not have these problems. However, existing Bayesian techniques lack scalability to large dataset and network sizes. In this work we present a novel scalable method for learning Bayesian neural networks, called probabilistic backpropagation (PBP). Similar to classical backpropagation, PBP works by computing a forward propagation of probabilities through the network and then doing a backward computation of gradients. A series of experiments on ten real-world datasets show that PBP is significantly faster than other techniques, while offering competitive predictive abilities. Our experiments also show that PBP provides accurate estimates of the posterior variance on the network weights.
References in corpus (3)
Cited by in corpus (69)
- Deep and Confident Prediction for Time Series at Uber
- Deep Learning: A Bayesian Perspective
- Dropout Inference in Bayesian Neural Networks with Alpha-divergences
- Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Space
- Quality of Uncertainty Quantification for Bayesian Neural Network Inference
- A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks
- Non-Intrusive Reduced-Order Modeling Using Uncertainty-Aware Deep Neural Networks and Proper Orthogonal Decomposition: Application to Flood Modeling
- Resource-efficient Deep Neural Networks for Automotive Radar Interference Mitigation
- Finite Versus Infinite Neural Networks: an Empirical Study
- Natural-Parameter Networks: A Class of Probabilistic Neural Networks
- Can You Trust This Prediction? Auditing Pointwise Reliability After Learning
- Qualitative Analysis of Monte Carlo Dropout
- 'In-Between' Uncertainty in Bayesian Neural Networks
- Uncertainty Decomposition in Bayesian Neural Networks with Latent Variables
- Bayesian Semisupervised Learning with Deep Generative Models
- PBODL : Parallel Bayesian Online Deep Learning for Click-Through Rate Prediction in Tencent Advertising System
- Bayesian Graph Convolutional Neural Networks using Node Copying
- Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions
- Estimating Model Uncertainty of Neural Networks in Sparse Information Form
- Benchmarking the Neural Linear Model for Regression
- Bayesian Graph Convolutional Neural Networks Using Non-Parametric Graph Learning
- Robust Deep Gaussian Processes
- Classification Confidence Estimation with Test-Time Data-Augmentation
- Radial and Directional Posteriors for Bayesian Neural Networks
- Non-Parametric Graph Learning for Bayesian Graph Neural Networks
- Variational Refinement for Importance Sampling Using the Forward Kullback-Leibler Divergence
- Evaluating Scalable Uncertainty Estimation Methods for DNN-Based Molecular Property Prediction
- RoNGBa: A Robustly Optimized Natural Gradient Boosting Training Approach with Leaf Number Clipping
- Mitigating Uncertainty in Document Classification
- Ensemble Model Patching: A Parameter-Efficient Variational Bayesian Neural Network
- Scalable Training of Inference Networks for Gaussian-Process Models
- Efficient Ensemble Model Generation for Uncertainty Estimation with Bayesian Approximation in Segmentation
- Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights
- Quantum Bayesian Neural Networks
- Applying SVGD to Bayesian Neural Networks for Cyclical Time-Series Prediction and Inference
- On Batch Normalisation for Approximate Bayesian Inference
- Generalized Bayesian Posterior Expectation Distillation for Deep Neural Networks
- Why have a Unified Predictive Uncertainty? Disentangling it using Deep Split Ensembles
- Efficient Evaluation-Time Uncertainty Estimation by Improved Distillation
- Sampling-Free Learning of Bayesian Quantized Neural Networks
- Adaptive Bayesian Linear Regression for Automated Machine Learning
- Efficient Online Bayesian Inference for Neural Bandits
- Uncertainty-Aware Model Adaptation for Unsupervised Cross-Domain Object Detection
- From Predictions to Decisions: Using Lookahead Regularization
- Uncertainty-aware Remaining Useful Life predictor
- Fully Bayesian Recurrent Neural Networks for Safe Reinforcement Learning
- Neural Likelihoods for Multi-Output Gaussian Processes
- Streaming Probabilistic Deep Tensor Factorization
- Prior Activation Distribution (PAD): A Versatile Representation to Utilize DNN Hidden Units
- Estimating Predictive Uncertainty Under Program Data Distribution Shift
- Stochastic Variational Inference via Upper Bound
- TyXe: Pyro-based Bayesian neural nets for Pytorch
- Efficient and Robust LiDAR-Based End-to-End Navigation
- Uncertainty Quantification in Deep Residual Neural Networks
- Assessing the Robustness of Bayesian Dark Knowledge to Posterior Uncertainty
- AutoCP: Automated Pipelines for Accurate Prediction Intervals
- Uncertainty Surrogates for Deep Learning
- Efficient Approximate Inference with Walsh-Hadamard Variational Inference
- Intrinsic uncertainties and where to find them
- A Bayesian Approach to Invariant Deep Neural Networks
- GP-ConvCNP: Better Generalization for Convolutional Conditional Neural Processes on Time Series Data
- Unifying Local and Global Change Detection in Dynamic Networks
- Why Calibration Error is Wrong Given Model Uncertainty: Using Posterior Predictive Checks with Deep Learning
- Robustness Against Outliers For Deep Neural Networks By Gradient Conjugate Priors
- Improving Uncertainty Calibration via Prior Augmented Data
- Uncertainty-Aware Multi-Modal Ensembling for Severity Prediction of Alzheimer's Dementia
- Beyond Marginal Uncertainty: How Accurately can Bayesian Regression Models Estimate Posterior Predictive Correlations?
- Bayesian Meta-Learning Through Variational Gaussian Processes
- Regularising Deep Networks with Deep Generative Models