Machine Learning Testing: Survey, Landscapes and Horizons

Abstract

This paper provides a comprehensive survey of Machine Learning Testing (ML testing) research. It covers 144 papers on testing properties (e.g., correctness, robustness, and fairness), testing components (e.g., the data, learning program, and framework), testing workflow (e.g., test generation and test evaluation), and application scenarios (e.g., autonomous driving, machine translation). The paper also analyses trends concerning datasets, research trends, and research focus, concluding with research challenges and promising research directions in ML testing.

References in corpus (42)

• Scikit-learn: Machine Learning in Python
• A Survey on Deep Learning in Medical Image Analysis
• Towards A Rigorous Science of Interpretable Machine Learning
• A Contextual-Bandit Approach to Personalized News Article Recommendation
• European Union regulations on algorithmic decision-making and a "right to explanation"
• Equality of Opportunity in Supervised Learning
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• Visualizing and Understanding Recurrent Networks
• Counterfactual Fairness
• On the Complexity of Best Arm Identification in Multi-Armed Bandit Models
• The Measure and Mismeasure of Fairness
• Metamorphic Testing: A New Approach for Generating Next Test Cases
• Guiding Deep Learning System Testing using Surprise Adequacy
• Fairness Testing: Testing Software for Discrimination
• Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
• Evaluating Robustness of Neural Networks with Mixed Integer Programming
• DLFuzz: Differential Fuzzing Testing of Deep Learning Systems
• Automated Directed Fairness Testing
• Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing
• Testing Deep Neural Networks
• On Formalizing Fairness in Prediction with Machine Learning
• Virtual to Real Reinforcement Learning for Autonomous Driving
• Verification for Machine Learning, Autonomy, and Neural Networks Survey
• AVFI: Fault Injection for Autonomous Vehicles
• Combinatorial Testing for Deep Learning Systems
• BoostClean: Automated Error Detection and Repair for Machine Learning
• Assessing the Local Interpretability of Machine Learning Models
• Symbolic Execution for Deep Neural Networks
• The Promise and Peril of Human Evaluation for Model Interpretability
• Detecting Adversarial Samples for Deep Neural Networks through Mutation Testing
• AlphaClean: Automatic Generation of Data Cleaning Pipelines
• Systematic Testing of Convolutional Neural Networks for Autonomous Driving
• DeepCruiser: Automated Guided Testing for Stateful Deep Learning Systems
• Kayotee: A Fault Injection-based System to Assess the Safety and Reliability of Autonomous Vehicles to Faults and Errors
• A Noise-Sensitivity-Analysis-Based Test Prioritization Technique for Deep Neural Networks
• Automated Test Generation to Detect Individual Discrimination in AI Models
• Global Robustness Evaluation of Deep Neural Networks with Provable Guarantees for the $L_0$ Norm
• Secure Deep Learning Engineering: A Software Quality Assurance Perspective
• Fairness-aware machine learning: a perspective
• Metamorphic Relation Based Adversarial Attacks on Differentiable Neural Computer
• Towards Testing of Deep Learning Systems with Training Set Reduction
• Test Selection for Deep Learning Systems

Cited by in corpus (16)

• Towards Guidelines for Assessing Qualities of Machine Learning Systems
• FakeSpotter: A Simple yet Robust Baseline for Spotting AI-Synthesized Fake Faces
• Operational Calibration: Debugging Confidence Errors for DNNs in the Field
• Software Engineering Practice in the Development of Deep Learning Applications
• Verifying Individual Fairness in Machine Learning Models
• CAGFuzz: Coverage-Guided Adversarial Generative Fuzzing Testing of Deep Learning Systems
• Automatic Testing and Improvement of Machine Translation
• Understanding the Nature of System-Related Issues in Machine Learning Frameworks: An Exploratory Study
• DeepSmartFuzzer: Reward Guided Test Generation For Deep Learning
• Quality Management of Machine Learning Systems
• Guiding the retraining of convolutional neural networks against adversarial inputs
• Structure-Invariant Testing for Machine Translation
• Risk Management Framework for Machine Learning Security
• Not Your Grandfathers Test Set: Reducing Labeling Effort for Testing
• ALT-MAS: A Data-Efficient Framework for Active Testing of Machine Learning Algorithms
• Testing Machine Translation via Referential Transparency