How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review

Abstract

Context: Machine Learning (ML) has been at the heart of many innovations over the past years. However, including it in so-called 'safety-critical' systems such as automotive or aeronautic has proven to be very challenging, since the shift in paradigm that ML brings completely changes traditional certification approaches. Objective: This paper aims to elucidate challenges related to the certification of ML-based safety-critical systems, as well as the solutions that are proposed in the literature to tackle them, answering the question 'How to Certify Machine Learning Based Safety-critical Systems?'. Method: We conduct a Systematic Literature Review (SLR) of research papers published between 2015 to 2020, covering topics related to the certification of ML systems. In total, we identified 217 papers covering topics considered to be the main pillars of ML certification: Robustness, Uncertainty, Explainability, Verification, Safe Reinforcement Learning, and Direct Certification. We analyzed the main trends and problems of each sub-field and provided summaries of the papers extracted. Results: The SLR results highlighted the enthusiasm of the community for this subject, as well as the lack of diversity in terms of datasets and type of models. It also emphasized the need to further develop connections between academia and industries to deepen the domain study. Finally, it also illustrated the necessity to build connections between the above mention main pillars that are for now mainly studied separately. Conclusion: We highlighted current efforts deployed to enable the certification of ML based software systems, and discuss some future research directions.

arXiv comment60 pages (92 pages with references and complements), submitted to a journal (Automated Software Engineering). Changes: Emphasizing difference traditional software engineering / ML approach. Adding Related Works, Threats to Validity and Complementary Materials. Adding a table listing papers reference for each section/subsections

References in corpus (74)

• Continuous control with deep reinforcement learning
• Explaining and Harnessing Adversarial Examples
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• European Union regulations on algorithmic decision-making and a "right to explanation"
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• An Introduction to Deep Reinforcement Learning
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker
• Opening the Black Box of Deep Neural Networks via Information
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems
• Metamorphic Testing: A New Approach for Generating Next Test Cases
• Certifying Some Distributional Robustness with Principled Adversarial Training
• DLFuzz: Differential Fuzzing Testing of Deep Learning Systems
• A General Framework for Uncertainty Estimation in Deep Learning
• Formal Security Analysis of Neural Networks using Symbolic Intervals
• Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation
• Scaling Out-of-Distribution Detection for Real-World Settings
• Efficient Formal Safety Analysis of Neural Networks
• Training robust neural networks using Lipschitz bounds
• A Game-Based Approximate Verification of Deep Neural Networks with Provable Guarantees
• Multivariate Confidence Calibration for Object Detection
• Safe Exploration in Finite Markov Decision Processes with Gaussian Processes
• Failing to Learn: Autonomously Identifying Perception Failures for Self-driving Cars
• Analyzing the Robustness of Nearest Neighbors to Adversarial Examples
• Bayesian Learning-Based Adaptive Control for Safety Critical Systems
• Unsolved Problems in ML Safety
• Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Control
• Systematic Mapping Study of Template-based Code Generation
• Towards Practical Verification of Machine Learning: The Case of Computer Vision Systems
• The Lyapunov Neural Network: Adaptive Stability Certification for Safe Learning of Dynamical Systems
• Provable Robustness of ReLU networks via Maximization of Linear Regions
• Analyzing Deep Neural Networks with Symbolic Propagation: Towards Higher Precision and Faster Verification
• Testing and verification of neural-network-based safety-critical control software: A systematic literature review
• Guaranteeing Safety for Neural Network-Based Aircraft Collision Avoidance Systems
• Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures
• Verifiably Safe Off-Model Reinforcement Learning
• Using Machine Learning Safely in Automotive Software: An Assessment and Adaption of Software Process Requirements in ISO 26262
• Formal Verification of Input-Output Mappings of Tree Ensembles
• Open the Black Box Data-Driven Explanation of Black Box Decision Systems
• Worst Cases Policy Gradients
• Systematic Testing of Convolutional Neural Networks for Autonomous Driving
• Robust Guarantees for Perception-Based Control
• Testing DNN Image Classifiers for Confusion & Bias Errors
• Safer Classification by Synthesis
• Can We Trust You? On Calibration of a Probabilistic Object Detector for Autonomous Driving
• Verifying Controllers Against Adversarial Examples with Bayesian Optimization
• Towards Safe Machine Learning for CPS: Infer Uncertainty from Training Data
• Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings
• Fast Neural Network Verification via Shadow Prices
• Learning Gaussian Processes by Minimizing PAC-Bayesian Generalization Bounds
• Experience-Based Heuristic Search: Robust Motion Planning with Deep Q-Learning
• Learning More Robust Features with Adversarial Training
• Lane-Merging Using Policy-based Reinforcement Learning and Post-Optimization
• Coverage Testing of Deep Learning Models using Dataset Characterization
• Verifying Aircraft Collision Avoidance Neural Networks Through Linear Approximations of Safe Regions
• Under the Hood of Neural Networks: Characterizing Learned Representations by Functional Neuron Populations and Network Ablations
• Verification of Neural Network Control Policy Under Persistent Adversarial Perturbation
• Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attack
• Certification of embedded systems based on Machine Learning: A survey
• Traceability of Deep Neural Networks
• Towards Robust Deep Neural Networks
• Playing it Safe: Adversarial Robustness with an Abstain Option
• Sampling-based Bayesian Inference with gradient uncertainty
• Improved Adversarial Robustness via Logit Regularization Methods
• Strength in Numbers: Trading-off Robustness and Computation via Adversarially-Trained Ensembles
• GraN: An Efficient Gradient-Norm Based Detector for Adversarial and Misclassified Examples
• Unsupervised Data Uncertainty Learning in Visual Retrieval Systems
• Brain-inspired reverse adversarial examples
• Risk-Aware Planning by Confidence Estimation using Deep Learning-Based Perception
• Safety-Aware Hardening of 3D Object Detection Neural Network Systems
• Affine Disentangled GAN for Interpretable and Robust AV Perception
• How Do You Act? An Empirical Study to Understand Behavior of Deep Reinforcement Learning Agents

Cited by in corpus (8)

• Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey
• DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks
• Optimality Principles in Spacecraft Neural Guidance and Control
• A Probabilistic Framework for Mutation Testing in Deep Neural Networks
• Perception Simplex: Verifiable Collision Avoidance in Autonomous Vehicles Amidst Obstacle Detection Faults
• Synergistic Perception and Control Simplex for Verifiable Safe Vertical Landing
• Bayes2IMC: In-Memory Computing for Bayesian Binary Neural Networks
• Policy Testing with MDPFuzz (Replicability Study)