Software Testing for Machine Learning

Abstract

Machine learning has become prevalent across a wide variety of applications. Unfortunately, machine learning has also shown to be susceptible to deception, leading to errors, and even fatal failures. This circumstance calls into question the widespread use of machine learning, especially in safety-critical applications, unless we are able to assure its correctness and trustworthiness properties. Software verification and testing are established technique for assuring such properties, for example by detecting errors. However, software testing challenges for machine learning are vast and profuse - yet critical to address. This summary talk discusses the current state-of-the-art of software testing for machine learning. More specifically, it discusses six key challenge areas for software testing of machine learning systems, examines current approaches to these challenges and highlights their limitations. The paper provides a research agenda with elaborated directions for making progress toward advancing the state-of-the-art on testing of machine learning.

References in corpus (15)

• Explaining and Harnessing Adversarial Examples
• Intriguing properties of neural networks
• Ensemble Adversarial Training: Attacks and Defenses
• Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation
• Adversarial Examples for Evaluating Reading Comprehension Systems
• Generating Adversarial Examples with Adversarial Networks
• Adversarial Examples: Attacks and Defenses for Deep Learning
• DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars
• Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing
• Testing Deep Neural Networks
• TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing
• Constructing Unrestricted Adversarial Examples with Generative Models
• Combinatorial Testing for Deep Learning Systems
• Interpretable Adversarial Perturbation in Input Embedding Space for Text
• DeepCruiser: Automated Guided Testing for Stateful Deep Learning Systems

Cited by in corpus (3)

• Machine Learning for Software Engineering: A Systematic Mapping
• Adversarial Deep Reinforcement Learning for Improving the Robustness of Multi-agent Autonomous Driving Policies
• Test and Evaluation Framework for Multi-Agent Systems of Autonomous Intelligent Agents