Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference

Abstract

Convolutional neural networks (CNNs) work well on large datasets. But labelled data is hard to collect, and in some applications larger amounts of data are not available. The problem then is how to use CNNs with small data -- as CNNs overfit quickly. We present an efficient Bayesian CNN, offering better robustness to over-fitting on small data than traditional approaches. This is by placing a probability distribution over the CNN's kernels. We approximate our model's intractable posterior with Bernoulli variational distributions, requiring no additional model parameters. On the theoretical side, we cast dropout network training as approximate inference in Bayesian neural networks. This allows us to implement our model using existing tools in deep learning with no increase in time complexity, while highlighting a negative result in the field. We show a considerable improvement in classification accuracy compared to standard techniques and improve on published state-of-the-art results for CIFAR-10.

12 pages, 3 figures, ICLR format, updated with reviewer comments

References in corpus (6)

• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Going Deeper with Convolutions
• Weight Uncertainty in Neural Networks
• Convolutional Kernel Networks
• Variational Bayesian Inference with Stochastic Search
• Improving the Gaussian Process Sparse Spectrum Approximation by Representing Uncertainty in Frequency Inputs

Cited by in corpus (150)

• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods
• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
• Deep Bayesian Active Learning with Image Data
• A Survey of Deep RL and IL for Autonomous Driving Policy Learning
• Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Towards Robust Evaluations of Continual Learning
• Exploiting Epistemic Uncertainty of Anatomy Segmentation for Anomaly Detection in Retinal OCT
• A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
• Bayesian-Deep-Learning Estimation of Earthquake Location from Single-Station Observations
• Indoor Scene Understanding in 2.5/3D for Autonomous Agents: A Survey
• Deep Confidence: A Computationally Efficient Framework for Calculating Reliable Errors for Deep Neural Networks
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• VIBNN: Hardware Acceleration of Bayesian Neural Networks
• Bayesian convolutional neural network based MRI brain extraction on nonhuman primates
• Cost-Effective Active Learning for Melanoma Segmentation
• Maximum-Entropy Adversarial Data Augmentation for Improved Generalization and Robustness
• Implicit Weight Uncertainty in Neural Networks
• Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference
• Parameters Estimation for the Cosmic Microwave Background with Bayesian Neural Networks
• Uncertainty Aware Semi-Supervised Learning on Graph Data
• Information fusion and machine learning for sensitivity analysis using physics knowledge and experimental data
• Introduction to Camera Pose Estimation with Deep Learning
• Trusted Multi-View Classification
• Estimating Depth from Monocular Images as Classification Using Deep Fully Convolutional Residual Networks
• Uncertainty-aware Self-training for Text Classification with Few Labels
• Exploiting Oxide Based Resistive RAM Variability for Bayesian Neural Network Hardware Design
• Transient-optimised real-bogus classification with Bayesian Convolutional Neural Networks -- sifting the GOTO candidate stream
• DeepIoT: Compressing Deep Neural Network Structures for Sensing Systems with a Compressor-Critic Framework
• Learning Uncertainty with Artificial Neural Networks for Improved Predictive Process Monitoring
• Uncertainty-driven Sanity Check: Application to Postoperative Brain Tumor Cavity Segmentation
• Calibration of Deep Probabilistic Models with Decoupled Bayesian Neural Networks
• Uncertainty-Aware Attention for Reliable Interpretation and Prediction
• VidLoc: A Deep Spatio-Temporal Model for 6-DoF Video-Clip Relocalization
• Propagating Uncertainty in Multi-Stage Bayesian Convolutional Neural Networks with Application to Pulmonary Nodule Detection
• Local Interpretable Model-agnostic Explanations of Bayesian Predictive Models via Kullback-Leibler Projections
• Qualitative Analysis of Monte Carlo Dropout
• Uncertainty Quantification for Molecular Property Predictions with Graph Neural Architecture Search
• Reducing Drift in Visual Odometry by Inferring Sun Direction Using a Bayesian Convolutional Neural Network
• Large-Scale Visual Active Learning with Deep Probabilistic Ensembles
• Parting with Illusions about Deep Active Learning
• The Dynamical Mass of the Coma Cluster from Deep Learning
• Approximate Bayesian Uncertainties on Deep Learning Dynamical Mass Estimates of Galaxy Clusters
• Adversarial Phenomenon in the Eyes of Bayesian Deep Learning
• ViPR: Visual-Odometry-aided Pose Regression for 6DoF Camera Localization
• GAlaxy Light profile convolutional neural NETworks (GaLNets). I. fast and accurate structural parameters for billion galaxy samples
• The Relevance of Bayesian Layer Positioning to Model Uncertainty in Deep Bayesian Active Learning
• Bayesian graph convolutional neural networks via tempered MCMC
• Deep learning with photosensor timing information as a background rejection method for the Cherenkov Telescope Array
• Regularizing Deep Networks with Semantic Data Augmentation
• Revisiting Bayesian Autoencoders with MCMC
• Bayesian Neural Networks With Maximum Mean Discrepancy Regularization
• Complex data labeling with deep learning methods: Lessons from fisheries acoustics
• Inhibited Softmax for Uncertainty Estimation in Neural Networks
• Learnable Bernoulli Dropout for Bayesian Deep Learning
• Modality Attention and Sampling Enables Deep Learning with Heterogeneous Marker Combinations in Fluorescence Microscopy
• Speech Recognition: Keyword Spotting Through Image Recognition
• A Decision-Theoretic Approach for Model Interpretability in Bayesian Framework
• Bayesian Graph Neural Networks with Adaptive Connection Sampling
• Active Learning for Deep Detection Neural Networks
• Sampling-Free Variational Inference of Bayesian Neural Networks by Variance Backpropagation
• Bayesian LSTMs in medicine
• Spatial Uncertainty Sampling for End-to-End Control
• Neural Simpletrons - Minimalistic Directed Generative Networks for Learning with Few Labels
• Robustly representing uncertainty in deep neural networks through sampling
• Information Dropout: Learning Optimal Representations Through Noisy Computation
• Semi-Supervised Domain Generalization with Stochastic StyleMatch
• Calibrating Deep Convolutional Gaussian Processes
• Mean-Field Neural ODEs via Relaxed Optimal Control
• Quicksilver: Fast Predictive Image Registration - a Deep Learning Approach
• A Variational Dirichlet Framework for Out-of-Distribution Detection
• RDeepSense: Reliable Deep Mobile Computing Models with Uncertainty Estimations
• Variational Inference to Measure Model Uncertainty in Deep Neural Networks
• Bayesian Neural Networks for Reversible Steganography
• Dense Uncertainty Estimation
• Advances in Deep Learning for Hyperspectral Image Analysis--Addressing Challenges Arising in Practical Imaging Scenarios
• Active Learning Solution on Distributed Edge Computing
• Integrating Uncertainty into Neural Network-based Speech Enhancement
• 3D Reconstruction in Canonical Co-ordinate Space from Arbitrarily Oriented 2D Images
• Deep Bayesian Active Learning, A Brief Survey on Recent Advances
• Generative Models For Deep Learning with Very Scarce Data
• On the use of uncertainty in classifying Aedes Albopictus mosquitoes
• Melody Generation for Pop Music via Word Representation of Musical Properties
• Fast and Reliable Probabilistic Face Embeddings in the Wild
• Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach
• Fast Predictive Multimodal Image Registration
• Multi-class Text Classification using BERT-based Active Learning
• Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability
• Bayesian posterior approximation with stochastic ensembles
• The Benefit of the Doubt: Uncertainty Aware Sensing for Edge Computing Platforms
• Exploring Uncertainty in Conditional Multi-Modal Retrieval Systems
• Known Unknowns: Uncertainty Quality in Bayesian Neural Networks
• Block Annotation: Better Image Annotation for Semantic Segmentation with Sub-Image Decomposition
• Walsh-Hadamard Variational Inference for Bayesian Deep Learning
• Efficient Ensemble Model Generation for Uncertainty Estimation with Bayesian Approximation in Segmentation
• Boosting Active Learning for Speech Recognition with Noisy Pseudo-labeled Samples
• Uncertainty-Aware Deep Classifiers using Generative Models
• Image segmentation of liver stage malaria infection with spatial uncertainty sampling
• Efficient Computation Reduction in Bayesian Neural Networks Through Feature Decomposition and Memorization
• Prediction intervals for Deep Neural Networks
• B-SCST: Bayesian Self-Critical Sequence Training for Image Captioning
• Cost-Effective Training of Deep CNNs with Active Model Adaptation
• Generative Self-training for Cross-domain Unsupervised Tagged-to-Cine MRI Synthesis
• Prior Activation Distribution (PAD): A Versatile Representation to Utilize DNN Hidden Units
• Variational Resampling Based Assessment of Deep Neural Networks under Distribution Shift
• On Modelling Label Uncertainty in Deep Neural Networks: Automatic Estimation of Intra-observer Variability in 2D Echocardiography Quality Assessment
• Structured Dropout Variational Inference for Bayesian Neural Networks
• Reliable Uncertainties for Bayesian Neural Networks using Alpha-divergences
• Efficient Uncertainty Estimation for Semantic Segmentation in Videos
• SkiNet: A Deep Learning Solution for Skin Lesion Diagnosis with Uncertainty Estimation and Explainability
• Bayesian Prediction of Future Street Scenes through Importance Sampling based Optimization
• Uncertainty-Aware Self-Supervised Target-Mass Grasping of Granular Foods
• Variational Bayes: A report on approaches and applications
• Deep Probabilistic Ensembles: Approximate Variational Inference through KL Regularization
• High-Order Stochastic Gradient Thermostats for Bayesian Learning of Deep Models
• Certainty Driven Consistency Loss on Multi-Teacher Networks for Semi-Supervised Learning
• Safety Enhancement for Deep Reinforcement Learning in Autonomous Separation Assurance
• Stochastic-Shield: A Probabilistic Approach Towards Training-Free Adversarial Defense in Quantized CNNs
• That Label's Got Style: Handling Label Style Bias for Uncertain Image Segmentation
• Bayesian Conditional GAN for MRI Brain Image Synthesis
• Probability Paths and the Structure of Predictions over Time
• An efficient scheme based on graph centrality to select nodes for training for effective learning
• Predictive Uncertainty through Quantization
• Cell Detection in Domain Shift Problem Using Pseudo-Cell-Position Heatmap
• Overestimation learning with guarantees
• Parameters Estimation from the 21 cm signal using Variational Inference
• Episode-Based Active Learning with Bayesian Neural Networks
• Deep Learning and Bayesian Deep Learning Based Gender Prediction in Multi-Scale Brain Functional Connectivity
• Long-Term On-Board Prediction of People in Traffic Scenes under Uncertainty
• Classifying States of Cooking Objects Using Convolutional Neural Network
• Probabilistic Spatial Analysis in Quantitative Microscopy with Uncertainty-Aware Cell Detection using Deep Bayesian Regression of Density Maps
• An Uncertainty-aware Hierarchical Probabilistic Network for Early Prediction, Quantification and Segmentation of Pulmonary Tumour Growth
• Sampling possible reconstructions of undersampled acquisitions in MR imaging
• Active Learning for Bayesian 3D Hand Pose Estimation
• Multiple Imputation for Biomedical Data using Monte Carlo Dropout Autoencoders
• Fast Predictive Image Registration
• GIM: Gaussian Isolation Machines
• High Frequency EEG Artifact Detection with Uncertainty via Early Exit Paradigm
• Loss Prediction: End-to-End Active Learning Approach For Speech Recognition
• Probabilistic Super-Resolution of Solar Magnetograms: Generating Many Explanations and Measuring Uncertainties
• Fundamental Issues Regarding Uncertainties in Artificial Neural Networks
• Uncertainty-Aware Data Aggregation for Deep Imitation Learning
• Analysis of ODE2VAE with Examples
• Improving Building Segmentation for Off-Nadir Satellite Imagery
• Trust but Verify: Assigning Prediction Credibility by Counterfactual Constrained Learning
• Uncertainty-Aware Label Refinement for Sequence Labeling
• Multi Agent System for Machine Learning Under Uncertainty in Cyber Physical Manufacturing System