Evaluating Bayesian Deep Learning Methods for Semantic Segmentation

Abstract

Deep learning has been revolutionary for computer vision and semantic segmentation in particular, with Bayesian Deep Learning (BDL) used to obtain uncertainty maps from deep models when predicting semantic classes. This information is critical when using semantic segmentation for autonomous driving for example. Standard semantic segmentation systems have well-established evaluation metrics. However, with BDL's rising popularity in computer vision we require new metrics to evaluate whether a BDL method produces better uncertainty estimates than another method. In this work we propose three such metrics to evaluate BDL models designed specifically for the task of semantic segmentation. We modify DeepLab-v3+, one of the state-of-the-art deep neural networks, and create its Bayesian counterpart using MC dropout and Concrete dropout as inference techniques. We then compare and test these two inference techniques on the well-known Cityscapes dataset using our suggested metrics. Our results provide new benchmarks for researchers to compare and evaluate their improved uncertainty quantification in pursuit of safer semantic segmentation.

arXiv commentUpdated baselines and numbers on concrete dropout

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Distributed Representations of Sentences and Documents
• End to End Learning for Self-Driving Cars
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• On Calibration of Modern Neural Networks
• Weight Uncertainty in Neural Networks
• Deep Learning using Linear Support Vector Machines
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• Variational Dropout and the Local Reparameterization Trick
• An Empirical Evaluation of Deep Learning on Highway Driving
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors

Cited by in corpus (36)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation
• A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applications
• SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation
• An Artificial Intelligence Framework for Bidding Optimization with Uncertainty in Multiple Frequency Reserve Markets
• QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results
• An Application-Driven Conceptualization of Corner Cases for Perception in Highly Automated Driving
• Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
• A Comparison of Uncertainty Estimation Approaches in Deep Learning Components for Autonomous Vehicle Applications
• Learnable Bernoulli Dropout for Bayesian Deep Learning
• Bayesian Graph Neural Networks with Adaptive Connection Sampling
• A Quantitative Comparison of Epistemic Uncertainty Maps Applied to Multi-Class Segmentation
• Deep Deterministic Uncertainty for Semantic Segmentation
• Contextual Dropout: An Efficient Sample-Dependent Dropout Module
• Dense Uncertainty Estimation
• Evidential Deep Learning for Open Set Action Recognition
• Perspective Aware Road Obstacle Detection
• One Versus all for deep Neural Network Incertitude (OVNNI) quantification
• Uncertainty aware audiovisual activity recognition using deep Bayesian variational inference
• Efficacy of Pixel-Level OOD Detection for Semantic Segmentation
• Evaluating Scalable Uncertainty Estimation Methods for DNN-Based Molecular Property Prediction
• Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes
• Uncertainty-Aware Physically-Guided Proxy Tasks for Unseen Domain Face Anti-spoofing
• A Bayesian Machine Learning Algorithm for Predicting ENSO Using Short Observational Time Series
• TRADI: Tracking deep neural network weight distributions for uncertainty estimation
• Approaching Neural Network Uncertainty Realism
• Improving Uncertainty-Error Correspondence in Deep Bayesian Medical Image Segmentation
• Alignment Attention by Matching Key and Query Distributions
• UNO: Uncertainty-aware Noisy-Or Multimodal Fusion for Unanticipated Input Degradation
• Probabilistic Semantic Segmentation Refinement by Monte Carlo Region Growing
• Bayesian Attention Modules
• Uncertainty quantification for White Matter Hyperintensity segmentation detects silent failures and improves automated Fazekas quantification
• Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model
• Consensus Synergizes with Memory: A Simple Approach for Anomaly Segmentation in Urban Scenes
• A Bayesian Neural Network based on Dropout Regulation