On-Demand Scenario Generation for Testing Automated Driving Systems

Abstract

The safety and reliability of Automated Driving Systems (ADS) are paramount, necessitating rigorous testing methodologies to uncover potential failures before deployment. Traditional testing approaches often prioritize either natural scenario sampling or safety-critical scenario generation, resulting in overly simplistic or unrealistic hazardous tests. In practice, the demand for natural scenarios (e.g., when evaluating the ADS's reliability in real-world conditions), critical scenarios (e.g., when evaluating safety in critical situations), or somewhere in between (e.g., when testing the ADS in regions with less civilized drivers) varies depending on the testing objectives. To address this issue, we propose the On-demand Scenario Generation (OSG) Framework, which generates diverse scenarios with varying risk levels. Achieving the goal of OSG is challenging due to the complexity of quantifying the criticalness and naturalness stemming from intricate vehicle-environment interactions, as well as the need to maintain scenario diversity across various risk levels. OSG learns from real-world traffic datasets and employs a Risk Intensity Regulator to quantitatively control the risk level. It also leverages an improved heuristic search method to ensure scenario diversity. We evaluate OSG on the Carla simulators using various ADSs. We verify OSG's ability to generate scenarios with different risk levels and demonstrate its necessity by comparing accident types across risk levels. With the help of OSG, we are now able to systematically and objectively compare the performance of different ADSs based on different risk levels.

20 pages, 9 figures. Accepted by FSE 2025

References in corpus (9)

• Experimental Comparison of Open Source Visual-Inertial-Based State Estimation Algorithms in the Underwater Domain
• GelTip: A Finger-shaped Optical Tactile Sensor for Robotic Manipulation
• Invisible for both Camera and LiDAR: Security of Multi-Sensor Fusion based Perception in Autonomous Driving Under Physical-World Attacks
• Augmented LiDAR Simulator for Autonomous Driving
• BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving Systems
• Adversarial Safety-Critical Scenario Generation using Naturalistic Human Driving Priors
• Towards Universal Physical Attacks On Cascaded Camera-Lidar 3D Object Detection Models
• Unsupervised Traffic Scene Generation with Synthetic 3D Scene Graphs
• Multimodal Safety-Critical Scenarios Generation for Decision-Making Algorithms Evaluation