A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks

Abstract

Evaluation of Bayesian deep learning (BDL) methods is challenging. We often seek to evaluate the methods' robustness and scalability, assessing whether new tools give `better' uncertainty estimates than old ones. These evaluations are paramount for practitioners when choosing BDL tools on-top of which they build their applications. Current popular evaluations of BDL methods, such as the UCI experiments, are lacking: Methods that excel with these experiments often fail when used in application such as medical or automotive, suggesting a pertinent need for new benchmarks in the field. We propose a new BDL benchmark with a diverse set of tasks, inspired by a real-world medical imaging application on \emph{diabetic retinopathy diagnosis}. Visual inputs (512x512 RGB images of retinas) are considered, where model uncertainty is used for medical pre-screening---i.e. to refer patients to an expert when model diagnosis is uncertain. Methods are then ranked according to metrics derived from expert-domain to reflect real-world use of model uncertainty in automated diagnosis. We develop multiple tasks that fall under this application, including out-of-distribution detection and robustness to distribution shift. We then perform a systematic comparison of well-tuned BDL techniques on the various tasks. From our comparison we conclude that some current techniques which solve benchmarks such as UCI `overfit' their uncertainty to the dataset---when evaluated on our benchmark these underperform in comparison to simpler baselines. The code for the benchmark, its baselines, and a simple API for evaluating new BDL tools are made available at https://github.com/oatml/bdl-benchmarks.

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Weight Uncertainty in Neural Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Uncertainty-Aware Reinforcement Learning for Collision Avoidance
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• Uncertainties in Parameters Estimated with Neural Networks: Application to Strong Gravitational Lensing
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation
• Functional Variational Bayesian Neural Networks
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• On the Importance of Strong Baselines in Bayesian Deep Learning
• Bayesian Deep Learning for Exoplanet Atmospheric Retrieval
• Variance Networks: When Expectation Does Not Meet Your Expectations
• Radial and Directional Posteriors for Bayesian Neural Networks

Cited by in corpus (16)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving
• Uncertainty Estimation Using a Single Deep Deterministic Neural Network
• What Are Bayesian Neural Network Posteriors Really Like?
• QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results
• Identifying Causal-Effect Inference Failure with Uncertainty-Aware Models
• Training independent subnetworks for robust prediction
• VI-PANN: Harnessing Transfer Learning and Uncertainty-Aware Variational Inference for Improved Generalization in Audio Pattern Recognition
• Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks
• Dangers of Bayesian Model Averaging under Covariate Shift
• Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning
• Benchmarking CNN on 3D Anatomical Brain MRI: Architectures, Data Augmentation and Deep Ensemble Learning
• On the Robustness of Pretraining and Self-Supervision for a Deep Learning-based Analysis of Diabetic Retinopathy
• Uncertainty estimation for classification and risk prediction on medical tabular data
• Deep Classifiers with Label Noise Modeling and Distance Awareness
• Law of Large Numbers for Bayesian two-layer Neural Network trained with Variational Inference