Paracosm: A Language and Tool for Testing Autonomous Driving Systems

Abstract

Systematic testing of autonomous vehicles operating in complex real-world scenarios is a difficult and expensive problem. We present Paracosm, a reactive language for writing test scenarios for autonomous driving systems. Paracosm allows users to programmatically describe complex driving situations with specific visual features, e.g., road layout in an urban environment, as well as reactive temporal behaviors of cars and pedestrians. Paracosm programs are executed on top of a game engine that provides realistic physics simulation and visual rendering. The infrastructure allows systematic exploration of the state space, both for visual features (lighting, shadows, fog) and for reactive interactions with the environment (pedestrians, other traffic). We define a notion of test coverage for Paracosm configurations based on combinatorial testing and low dispersion sequences. Paracosm comes with an automatic test case generator that uses random sampling for discrete parameters and deterministic quasi-Monte Carlo generation for continuous parameters. Through an empirical evaluation, we demonstrate the modeling and testing capabilities of Paracosm on a suite of autonomous driving systems implemented using deep neural networks developed in research and education. We show how Paracosm can expose incorrect behaviors or degraded performance.

arXiv commentTechnical Report to complement paper at FASE 2021

References in corpus (5)

• End to End Learning for Self-Driving Cars
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• Guiding Deep Learning System Testing using Surprise Adequacy
• Exploring the Space of Black-box Attacks on Deep Neural Networks
• Vehicle Platooning Simulations with Functional Reactive Programming

Cited by in corpus (7)

• Software Engineering for AI-Based Systems: A Survey
• Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environments
• GzScenic: Automatic Scene Generation for Gazebo Simulator
• Can Offline Testing of Deep Neural Networks Replace Their Online Testing?
• Autonomics: In Search of a Foundation for Next Generation Autonomous Systems
• Synthesis of Static Test Environments for Observing Sequence-like Behaviors in Autonomous Systems
• A Customizable Dynamic Scenario Modeling and Data Generation Platform for Autonomous Driving