The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation

Abstract

Deep learning has enabled impressive progress in the accuracy of semantic segmentation. Yet, the ability to estimate uncertainty and detect failure is key for safety-critical applications like autonomous driving. Existing uncertainty estimates have mostly been evaluated on simple tasks, and it is unclear whether these methods generalize to more complex scenarios. We present Fishyscapes, the first public benchmark for uncertainty estimation in a real-world task of semantic segmentation for urban driving. It evaluates pixel-wise uncertainty estimates towards the detection of anomalous objects in front of the vehicle. We~adapt state-of-the-art methods to recent semantic segmentation models and compare approaches based on softmax confidence, Bayesian learning, and embedding density. Our results show that anomaly detection is far from solved even for ordinary situations, while our benchmark allows measuring advancements beyond the state-of-the-art.

arXiv commentbechmark website: https://fishyscapes.com/

References in corpus (34)

• Explaining and Harnessing Adversarial Examples
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• On Calibration of Modern Neural Networks
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Semantic Foggy Scene Understanding with Synthetic Data
• NICE: Non-linear Independent Components Estimation
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Density estimation using Real NVP
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• Deep Continuous Fusion for Multi-Sensor 3D Object Detection
• Deep Anomaly Detection Using Geometric Transformations
• Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
• Learning Confidence for Out-of-Distribution Detection in Neural Networks
• Predictive Uncertainty Estimation via Prior Networks
• A General Framework for Uncertainty Estimation in Deep Learning
• Uncertainty Estimation Using a Single Deep Deterministic Neural Network
• To Trust Or Not To Trust A Classifier
• A2D2: Audi Autonomous Driving Dataset
• TensorFlow Distributions
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Glow: Generative Flow with Invertible 1x1 Convolutions
• Deep Ordinal Regression Network for Monocular Depth Estimation
• Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation
• WAIC, but Why? Generative Ensembles for Robust Anomaly Detection
• Evaluating Bayesian Deep Learning Methods for Semantic Segmentation
• Generative Probabilistic Novelty Detection with Adversarial Autoencoders
• Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples
• Distance-based Confidence Score for Neural Network Classifiers
• Do Deep Generative Models Know What They Don't Know?
• Modeling Uncertainty by Learning a Hierarchy of Deep Neural Connections

Cited by in corpus (30)

• Anomaly Detection in Autonomous Driving: A Survey
• Semantics for Robotic Mapping, Perception and Interaction: A Survey
• An Application-Driven Conceptualization of Corner Cases for Perception in Highly Automated Driving
• Perception Datasets for Anomaly Detection in Autonomous Driving: A Survey
• Learnable Bernoulli Dropout for Bayesian Deep Learning
• Learning Densities in Feature Space for Reliable Segmentation of Indoor Scenes
• Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
• Ad-datasets: a meta-collection of data sets for autonomous driving
• Two Video Data Sets for Tracking and Retrieval of Out of Distribution Objects
• Multimodal Detection of Unknown Objects on Roads for Autonomous Driving
• An Outlier Exposure Approach to Improve Visual Anomaly Detection Performance for Mobile Robots
• A Quantitative Comparison of Epistemic Uncertainty Maps Applied to Multi-Class Segmentation
• Learning Uncertainty For Safety-Oriented Semantic Segmentation In Autonomous Driving
• The Hidden Uncertainty in a Neural Networks Activations
• Hybrid Open-set Segmentation with Synthetic Negative Data
• Segmenting Known Objects and Unseen Unknowns without Prior Knowledge
• Weakly supervised training of universal visual concepts for multi-domain semantic segmentation
• Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes
• Class Anchor Clustering: a Loss for Distance-based Open Set Recognition
• AnoVox: A Benchmark for Multimodal Anomaly Detection in Autonomous Driving
• Conditioning Latent-Space Clusters for Real-World Anomaly Classification
• Fast Uncertainty Quantification for Deep Object Pose Estimation
• Experiments on Anomaly Detection in Autonomous Driving by Forward-Backward Style Transfers
• UNO: Uncertainty-aware Noisy-Or Multimodal Fusion for Unanticipated Input Degradation
• SupLID: Geometrical Guidance for Out-of-Distribution Detection in Semantic Segmentation
• Segmenting Objectiveness and Task-awareness Unknown Region for Autonomous Driving
• Active Learning for UAV-based Semantic Mapping
• Compressing Sensor Data for Remote Assistance of Autonomous Vehicles using Deep Generative Models
• Self-Improving Semantic Perception for Indoor Localisation
• Consensus Synergizes with Memory: A Simple Approach for Anomaly Segmentation in Urban Scenes