Towards Safe Machine Learning for CPS: Infer Uncertainty from Training Data

Abstract

Machine learning (ML) techniques are increasingly applied to decision-making and control problems in Cyber-Physical Systems among which many are safety-critical, e.g., chemical plants, robotics, autonomous vehicles. Despite the significant benefits brought by ML techniques, they also raise additional safety issues because 1) most expressive and powerful ML models are not transparent and behave as a black box and 2) the training data which plays a crucial role in ML safety is usually incomplete. An important technique to achieve safety for ML models is "Safe Fail", i.e., a model selects a reject option and applies the backup solution, a traditional controller or a human operator for example, when it has low confidence in a prediction. Data-driven models produced by ML algorithms learn from training data, and hence they are only as good as the examples they have learnt. As pointed in [17], ML models work well in the "training space" (i.e., feature space with sufficient training data), but they could not extrapolate beyond the training space. As observed in many previous studies, a feature space that lacks training data generally has a much higher error rate than the one that contains sufficient training samples [31]. Therefore, it is essential to identify the training space and avoid extrapolating beyond the training space. In this paper, we propose an efficient Feature Space Partitioning Tree (FSPT) to address this problem. Using experiments, we also show that, a strong relationship exists between model performance and FSPT score.

arXiv commentPublication rights licensed to ACM

References in corpus (10)

• Auto-Encoding Variational Bayes
• End to End Learning for Self-Driving Cars
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Deep Learning: A Critical Appraisal
• "Why Should I Trust You?": Explaining the Predictions of Any Classifier
• Automated Directed Fairness Testing
• Variational Bayesian Inference with Stochastic Search
• An Analysis of ISO 26262: Using Machine Learning Safely in Automotive Software
• On-line predictive linear regression
• On-line predictive linear regression

Cited by in corpus (5)

• How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
• Embedded out-of-distribution detection on an autonomous robot platform
• Real-time Out-of-distribution Detection in Learning-Enabled Cyber-Physical Systems
• Assurance Monitoring of Cyber-Physical Systems with Machine Learning Components
• Does the dataset meet your expectations? Explaining sample representation in image data