A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference

Abstract

Artificial Neural Networks are connectionist systems that perform a given task by learning on examples without having prior knowledge about the task. This is done by finding an optimal point estimate for the weights in every node. Generally, the network using point estimates as weights perform well with large datasets, but they fail to express uncertainty in regions with little or no data, leading to overconfident decisions. In this paper, Bayesian Convolutional Neural Network (BayesCNN) using Variational Inference is proposed, that introduces probability distribution over the weights. Furthermore, the proposed BayesCNN architecture is applied to tasks like Image Classification, Image Super-Resolution and Generative Adversarial Networks. The results are compared to point-estimates based architectures on MNIST, CIFAR-10 and CIFAR-100 datasets for Image CLassification task, on BSD300 dataset for Image Super Resolution task and on CIFAR10 dataset again for Generative Adversarial Network task. BayesCNN is based on Bayes by Backprop which derives a variational approximation to the true posterior. We, therefore, introduce the idea of applying two convolutional operations, one for the mean and one for the variance. Our proposed method not only achieves performances equivalent to frequentist inference in identical architectures but also incorporate a measurement for uncertainties and regularisation. It further eliminates the use of dropout in the model. Moreover, we predict how certain the model prediction is based on the epistemic and aleatoric uncertainties and empirically show how the uncertainty can decrease, allowing the decisions made by the network to become more deterministic as the training accuracy increases. Finally, we propose ways to prune the Bayesian architecture and to make it more computational and time effective.

arXiv admin note: text overlap with arXiv:1506.02158, arXiv:1703.04977 by other authors

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Improving neural networks by preventing co-adaptation of feature detectors
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Weight Uncertainty in Neural Networks
• Compressing Deep Convolutional Networks using Vector Quantization
• Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
• Variational Dropout and the Local Reparameterization Trick
• Learning the Number of Neurons in Deep Networks
• Bayesian Recurrent Neural Networks
• Exploring Sparsity in Recurrent Neural Networks
• The Power of Sparsity in Convolutional Neural Networks
• Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference
• Noise Contrastive Priors for Functional Uncertainty
• Stochastic Backpropagation through Mixture Density Distributions
• Variance Networks: When Expectation Does Not Meet Your Expectations

Cited by in corpus (29)

• Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference
• Continual Deep Learning by Functional Regularisation of Memorable Past
• Bayesian neural networks via MCMC: a Python-based tutorial
• ProbAct: A Probabilistic Activation Function for Deep Neural Networks
• Bayesian Neural Networks With Maximum Mean Discrepancy Regularization
• Bayesian Neural Ordinary Differential Equations
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• LUVLi Face Alignment: Estimating Landmarks' Location, Uncertainty, and Visibility Likelihood
• Towards calibrated and scalable uncertainty representations for neural networks
• Bayesian Convolutional Neural Networks for Limited Data Hyperspectral Remote Sensing Image Classification
• Recurrent Localization Networks applied to the Lippmann-Schwinger Equation
• Deep Bayesian Gaussian Processes for Uncertainty Estimation in Electronic Health Records
• A Financial Service Chatbot based on Deep Bidirectional Transformers
• W2WNet: a two-module probabilistic Convolutional Neural Network with embedded data cleansing functionality
• Shift-BNN: Highly-Efficient Probabilistic Bayesian Neural Network Training via Memory-Friendly Pattern Retrieving
• Bayesian Neural Networks at Scale: A Performance Analysis and Pruning Study
• Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability
• Sampling-Free Learning of Bayesian Quantized Neural Networks
• An evaluation of the LLC4320 global ocean simulation based on the submesoscale structure of modeled sea surface temperature fields
• Estimation of redshift and associated uncertainty of Fermi/LAT extra-galactic sources with Deep Learning
• Variational Resampling Based Assessment of Deep Neural Networks under Distribution Shift
• Deep inference of simulated strong lenses in ground-based surveys
• Performance Measurement for Deep Bayesian Neural Network
• Explanation and Use of Uncertainty Quantified by Bayesian Neural Network Classifiers for Breast Histopathology Images
• Improved Image Matting via Real-time User Clicks and Uncertainty Estimation
• MCU-Net: A framework towards uncertainty representations for decision support system patient referrals in healthcare contexts
• A Bayesian Approach to Invariant Deep Neural Networks
• Multi Agent System for Machine Learning Under Uncertainty in Cyber Physical Manufacturing System
• Trust but Verify: Assigning Prediction Credibility by Counterfactual Constrained Learning