Engineering problems in machine learning systems

Abstract

Fatal accidents are a major issue hindering the wide acceptance of safety-critical systems that employ machine learning and deep learning models, such as automated driving vehicles. In order to use machine learning in a safety-critical system, it is necessary to demonstrate the safety and security of the system through engineering processes. However, thus far, no such widely accepted engineering concepts or frameworks have been established for these systems. The key to using a machine learning model in a deductively engineered system is decomposing the data-driven training of machine learning models into requirement, design, and verification, particularly for machine learning models used in safety-critical systems. Simultaneously, open problems and relevant technical fields are not organized in a manner that enables researchers to select a theme and work on it. In this study, we identify, classify, and explore the open problems in engineering (safety-critical) machine learning systems --- that is, in terms of requirement, design, and verification of machine learning models and systems --- as well as discuss related works and research directions, using automated driving vehicles as an example. Our results show that machine learning models are characterized by a lack of requirements specification, lack of design specification, lack of interpretability, and lack of robustness. We also perform a gap analysis on a conventional system quality standard SQuARE with the characteristics of machine learning models to study quality models for machine learning systems. We find that a lack of requirements specification and lack of robustness have the greatest impact on conventional quality models.

arXiv comment20 pages, 4 figures, 4 tables

References in corpus (11)

• Sequence to Sequence Learning with Neural Networks
• Intriguing properties of neural networks
• Playing Atari with Deep Reinforcement Learning
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• Understanding deep learning requires rethinking generalization
• Characterizing Concept Drift
• Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
• Towards Proving the Adversarial Robustness of Deep Neural Networks
• An Analysis of ISO 26262: Using Machine Learning Safely in Automotive Software
• Runtime Monitoring Neuron Activation Patterns
• Towards Dependability Metrics for Neural Networks

Cited by in corpus (1)

• Causal Models in Requirement Specifications for Machine Learning: A vision