DeepRoad: GAN-based Metamorphic Autonomous Driving System Testing

Abstract

While Deep Neural Networks (DNNs) have established the fundamentals of DNN-based autonomous driving systems, they may exhibit erroneous behaviors and cause fatal accidents. To resolve the safety issues of autonomous driving systems, a recent set of testing techniques have been designed to automatically generate test cases, e.g., new input images transformed from the original ones. Unfortunately, many such generated input images often render inferior authenticity, lacking accurate semantic information of the driving scenes and hence compromising the resulting efficacy and reliability. In this paper, we propose DeepRoad, an unsupervised framework to automatically generate large amounts of accurate driving scenes to test the consistency of DNN-based autonomous driving systems across different scenes. In particular, DeepRoad delivers driving scenes with various weather conditions (including those with rather extreme conditions) by applying the Generative Adversarial Networks (GANs) along with the corresponding real-world weather scenes. Moreover, we have implemented DeepRoad to test three well-recognized DNN-based autonomous driving systems. Experimental results demonstrate that DeepRoad can detect thousands of behavioral inconsistencies in these systems.

arXiv comment7 pages

References in corpus (5)

• End to End Learning for Self-Driving Cars
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• Unsupervised Image-to-Image Translation Networks
• DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars
• Semantic-aware Grad-GAN for Virtual-to-Real Urban Scene Adaption

Cited by in corpus (20)

• Testing Deep Neural Networks
• Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges
• A Survey of Safety and Trustworthiness of Deep Neural Networks: Verification, Testing, Adversarial Attack and Defence, and Interpretability
• Guidance on the Assurance of Machine Learning in Autonomous Systems (AMLAS)
• DeepBillboard: Systematic Physical-World Testing of Autonomous Driving Systems
• A Noise-Sensitivity-Analysis-Based Test Prioritization Technique for Deep Neural Networks
• Is it Safe to Drive? An Overview of Factors, Challenges, and Datasets for Driveability Assessment in Autonomous Driving
• A Little Fog for a Large Turn
• Deep Learning in the Automotive Industry: Recent Advances and Application Examples
• PhysGAN: Generating Physical-World-Resilient Adversarial Examples for Autonomous Driving
• DeepSmartFuzzer: Reward Guided Test Generation For Deep Learning
• Towards Structured Evaluation of Deep Neural Network Supervisors
• Towards Adversarial Configurations for Software Product Lines
• Slimmable Generative Adversarial Networks
• Explaining Image Classifiers using Statistical Fault Localization
• Structure-Invariant Testing for Machine Translation
• Automated Testing for Deep Learning Systems with Differential Behavior Criteria
• Reliability Validation of Learning Enabled Vehicle Tracking
• Towards Quality Assurance of Software Product Lines with Adversarial Configurations
• Testing Machine Translation via Referential Transparency