Simulator Ensembles for Trustworthy Autonomous Driving Systems Testing

Abstract

Scenario-based testing with driving simulators is extensively used to identify failing conditions of automated driving assistance systems (ADAS). However, existing studies have shown that repeated test execution in the same as well as in distinct simulators can yield different outcomes, which can be attributed to sources of flakiness or different implementations of the physics. In this paper, we present MultiSim, a novel approach to multi-simulation ADAS testing based on a search-based testing approach that leverages an ensemble of simulators to identify failure-inducing, simulator-agnostic test scenarios. During the search, each scenario is evaluated jointly on multiple simulators. Scenarios that produce consistent results across simulators are prioritized for further exploration, while those that fail on only a subset of simulators are given less priority, as they may reflect simulator-specific issues rather than generalizable failures. Our empirical study, which involves testing three lane-keeping ADAS on different pairs of three widely used simulators, demonstrates that MultiSim outperforms single-simulator testing by achieving, on average, a higher rate of simulator-agnostic failures by 66%. Compared to a state-of-the-art multi-simulator approach that combines the outcome of independent test generation campaigns obtained in different simulators, MultiSim identifies, on average, up to 3.4X more simulator-agnostic failing tests and higher failure rates. To avoid the costly execution of test inputs on which simulators disagree, we propose to predict simulator disagreements and bypass test executions. Our results show that utilizing a surrogate model during the search retains the average number of valid failures and also improves efficiency. Our findings indicate that combining an ensemble of simulators is a promising approach for the automated cross-replication in ADAS testing.

References in corpus (12)

• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• How to Evaluate Solutions in Pareto-based Search-Based Software Engineering? A Critical Review and Methodological Guidance
• Mind the Gap! A Study on the Transferability of Virtual vs Physical-world Testing of Autonomous Driving Systems
• DriveFuzz: Discovering Autonomous Driving Bugs through Driving Quality-Guided Fuzzing
• Robotics Software Engineering: A Perspective from the Service Robotics Domain
• Choose Your Simulator Wisely: A Review on Open-source Simulators for Autonomous Driving
• BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving Systems
• Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing
• Boundary State Generation for Testing and Improvement of Autonomous Driving Systems
• Autonomous Driving Small-Scale Cars: A Survey of Recent Development
• Reinforcement Learning Informed Evolutionary Search for Autonomous Systems Testing
• A Survey on Small-Scale Testbeds for Connected and Automated Vehicles and Robot Swarms