A Safety Framework for Critical Systems Utilising Deep Neural Networks

Abstract

Increasingly sophisticated mathematical modelling processes from Machine Learning are being used to analyse complex data. However, the performance and explainability of these models within practical critical systems requires a rigorous and continuous verification of their safe utilisation. Working towards addressing this challenge, this paper presents a principled novel safety argument framework for critical systems that utilise deep neural networks. The approach allows various forms of predictions, e.g., future reliability of passing some demands, or confidence on a required reliability level. It is supported by a Bayesian analysis using operational data and the recent verification and validation techniques for deep learning. The prediction is conservative -- it starts with partial prior knowledge obtained from lifecycle activities and then determines the worst-case prediction. Open challenges are also identified.

Accepted by SafeComp2020

References in corpus (9)

• Deep Learning with Differential Privacy
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• An approach to reachability analysis for feed-forward ReLU neural networks
• Gradient Descent Finds Global Minima of Deep Neural Networks
• A Game-Based Approximate Verification of Deep Neural Networks with Provable Guarantees
• Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges
• Assessing the Safety and Reliability of Autonomous Vehicles from Road Testing
• Using Machine Learning Safely in Automotive Software: An Assessment and Adaption of Software Process Requirements in ISO 26262
• Towards Quantification of Assurance for Learning-enabled Components

Cited by in corpus (12)

• Assessing Safety-Critical Systems from Operational Testing: A Study on Autonomous Vehicles
• Reliability Assessment and Safety Arguments for Machine Learning Components in System Assurance
• Detecting Operational Adversarial Examples for Reliable Deep Learning
• Assessing the Reliability of Deep Learning Classifiers Through Robustness Evaluation and Operational Profiles
• Adversarial Robustness of Deep Learning: Theory, Algorithms, and Applications
• Abstraction and Symbolic Execution of Deep Neural Networks with Bayesian Approximation of Hidden Features
• Provably-Robust Runtime Monitoring of Neuron Activation Patterns
• Tutorials on Testing Neural Networks
• Safety Case Templates for Autonomous Systems
• Safety Metrics for Semantic Segmentation in Autonomous Driving
• Logically Sound Arguments for the Effectiveness of ML Safety Measures
• Is the Rush to Machine Learning Jeopardizing Safety? Results of a Survey