Neural Bridge Sampling for Evaluating Safety-Critical Autonomous Systems
arXiv:2008.10581
Abstract
Learning-based methodologies increasingly find applications in safety-critical domains like autonomous driving and medical robotics. Due to the rare nature of dangerous events, real-world testing is prohibitively expensive and unscalable. In this work, we employ a probabilistic approach to safety evaluation in simulation, where we are concerned with computing the probability of dangerous events. We develop a novel rare-event simulation method that combines exploration, exploitation, and optimization techniques to find failure modes and estimate their rate of occurrence. We provide rigorous guarantees for the performance of our method in terms of both statistical and computational efficiency. Finally, we demonstrate the efficacy of our approach on a variety of scenarios, illustrating its usefulness as a tool for rapid sensitivity analysis and model comparison that are essential to developing and testing safety-critical autonomous systems.
arXiv commentNeurIPS 2020
References in corpus (21)
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Variational Inference with Normalizing Flows
- A Conceptual Introduction to Hamiltonian Monte Carlo
- On a Formal Model of Safe and Scalable Self-driving Cars
- Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net
- Normalizing Flows for Probabilistic Modeling and Inference
- Certifying Some Distributional Robustness with Principled Adversarial Training
- Anomaly Detection with Density Estimation
- Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
- WAIC, but Why? Generative Ensembles for Robust Anomaly Detection
- Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithm
- Rapid Mixing of Hamiltonian Monte Carlo on Strongly Log-Concave Distributions
- NeuTra-lizing Bad Geometry in Hamiltonian Monte Carlo Using Neural Transport
- Efficient Black-box Assessment of Autonomous Vehicle Safety
- Estimating Ratios of Normalizing Constants Using Linked Importance Sampling
- Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures
- Spherical Hamiltonian Monte Carlo for Constrained Target Distributions
- Fast mixing of Metropolized Hamiltonian Monte Carlo: Benefits of multi-step gradients
- A Statistical Approach to Assessing Neural Network Robustness
- Automatic Testing With Reusable Adversarial Agents
- In-silico Risk Analysis of Personalized Artificial Pancreas Controllers via Rare-event Simulation