Testing and verification of neural-network-based safety-critical control software: A systematic literature review

Abstract

Context: Neural Network (NN) algorithms have been successfully adopted in a number of Safety-Critical Cyber-Physical Systems (SCCPSs). Testing and Verification (T&V) of NN-based control software in safety-critical domains are gaining interest and attention from both software engineering and safety engineering researchers and practitioners. Objective: With the increase in studies on the T&V of NN-based control software in safety-critical domains, it is important to systematically review the state-of-the-art T&V methodologies, to classify approaches and tools that are invented, and to identify challenges and gaps for future studies. Method: We retrieved 950 papers on the T&V of NN-based Safety-Critical Control Software (SCCS). To reach our result, we filtered 83 primary papers published between 2001 and 2018, applied the thematic analysis approach for analyzing the data extracted from the selected papers, presented the classification of approaches, and identified challenges. Conclusion: The approaches were categorized into five high-order themes: assuring robustness of NNs, assuring safety properties of NN-based control software, improving the failure resilience of NNs, measuring and ensuring test completeness, and improving the interpretability of NNs. From the industry perspective, improving the interpretability of NNs is a crucial need in safety-critical applications. We also investigated nine safety integrity properties within four major safety lifecycle phases to investigate the achievement level of T&V goals in IEC 61508-3. Results show that correctness, completeness, freedom from intrinsic faults, and fault tolerance have drawn most attention from the research community. However, little effort has been invested in achieving repeatability; no reviewed study focused on precisely defined testing configuration or on defense against common cause failure.

arXiv commentThis paper had been submitted to Journal of Information and Software Technology on April 20, 2019,Revised 5 December 2019, Accepted 6 March 2020, Available online 7 March 2020

References in corpus (36)

• Distilling the Knowledge in a Neural Network
• Continuous control with deep reinforcement learning
• End to End Learning for Self-Driving Cars
• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• Interpretable Explanations of Black Boxes by Meaningful Perturbation
• SmoothGrad: removing noise by adding noise
• Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices
• Towards Deep Neural Network Architectures Robust to Adversarial Examples
• On a Formal Model of Safe and Scalable Self-driving Cars
• Understanding Adversarial Training: Increasing Local Stability of Neural Nets through Robust Optimization
• Detecting Adversarial Samples from Artifacts
• DLFuzz: Differential Fuzzing Testing of Deep Learning Systems
• Learning how to explain neural networks: PatternNet and PatternAttribution
• CARLA: An Open Urban Driving Simulator
• Distilling a Neural Network Into a Soft Decision Tree
• Synthesizing the preferred inputs for neurons in neural networks via deep generator networks
• Software Engineering Challenges of Deep Learning
• Local Rule-Based Explanations of Black Box Decision Systems
• Towards Proving the Adversarial Robustness of Deep Neural Networks
• A Game-Based Approximate Verification of Deep Neural Networks with Provable Guarantees
• Distilling Knowledge from Deep Networks with Applications to Healthcare Domain
• Interpretability via Model Extraction
• Improving the Robustness of Deep Neural Networks via Stability Training
• Extending Defensive Distillation
• Output Range Analysis for Deep Neural Networks
• Reachable Set Computation and Safety Verification for Neural Networks with ReLU Activations
• DeepSafe: A Data-driven Approach for Checking Adversarial Robustness in Neural Networks
• Experimental Resilience Assessment of An Open-Source Driving Agent
• TreeView: Peeking into Deep Neural Networks Via Feature-Space Partitioning
• Toward Scalable Verification for Safety-Critical Deep Networks
• Interpreting Deep Classifier by Visual Distillation of Dark Knowledge
• A Study of Deep Learning Robustness Against Computation Failures
• Verification of Binarized Neural Networks via Inter-Neuron Factoring

Cited by in corpus (6)

• How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
• A Systematic Literature Review on Safety of the Intended Functionality for Automated Driving Systems
• Policy Testing with MDPFuzz (Replicability Study)
• Local Concept Embeddings for Analysis of Concept Distributions in Vision DNN Feature Spaces
• Beyond Confidence: Adaptive Abstention in Dual-Threshold Conformal Prediction for Autonomous System Perception
• Concepts in Testing of Autonomous Systems: Academic Literature and Industry Practice