Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Control

Abstract

A rise in popularity of Deep Neural Networks (DNNs), attributed to more powerful GPUs and widely available datasets, has seen them being increasingly used within safety-critical domains. One such domain, self-driving, has benefited from significant performance improvements, with millions of miles having been driven with no human intervention. Despite this, crashes and erroneous behaviours still occur, in part due to the complexity of verifying the correctness of DNNs and a lack of safety guarantees. In this paper, we demonstrate how quantitative measures of uncertainty can be extracted in real-time, and their quality evaluated in end-to-end controllers for self-driving cars. To this end we utilise a recent method for gathering approximate uncertainty information from DNNs without changing the network's architecture. We propose evaluation techniques for the uncertainty on two separate architectures which use the uncertainty to predict crashes up to five seconds in advance. We find that mutual information, a measure of uncertainty in classification networks, is a promising indicator of forthcoming crashes.

arXiv comment7 pages, 6 figures

References in corpus (4)

• Improving neural networks by preventing co-adaptation of feature detectors
• End to End Learning for Self-Driving Cars
• Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car
• End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perception

Cited by in corpus (22)

• How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
• Run-Time Monitoring of Machine Learning for Robotic Perception: A Survey of Emerging Trends
• Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving
• Practical Solutions for Machine Learning Safety in Autonomous Vehicles
• Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning
• A Comparison of Uncertainty Estimation Approaches in Deep Learning Components for Autonomous Vehicle Applications
• On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation
• Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification
• Predicting Model Failure using Saliency Maps in Autonomous Driving Systems
• A Quantitative Comparison of Epistemic Uncertainty Maps Applied to Multi-Class Segmentation
• Towards Quantification of Assurance for Learning-enabled Components
• Dynamic Bottleneck for Robust Self-Supervised Exploration
• Learning Uncertainty For Safety-Oriented Semantic Segmentation In Autonomous Driving
• Generalizing from a few environments in safety-critical reinforcement learning
• Fast Predictive Uncertainty for Classification with Bayesian Deep Networks
• Towards Dependable Autonomous Systems Based on Bayesian Deep Learning Components
• Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks
• Understanding the Under-Coverage Bias in Uncertainty Estimation
• Deep Neural Network Perception Models and Robust Autonomous Driving Systems
• Quantifying Epistemic Uncertainty in Deep Learning
• Efficient and Robust LiDAR-Based End-to-End Navigation
• Robust Vision-Based Cheat Detection in Competitive Gaming