DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks

Abstract

Deep neural networks (DNNs) are widely used in various application domains such as image processing, speech recognition, and natural language processing. However, testing DNN models may be challenging due to the complexity and size of their input domain. Particularly, testing DNN models often requires generating or exploring large unlabeled datasets. In practice, DNN test oracles, which identify the correct outputs for inputs, often require expensive manual effort to label test data, possibly involving multiple experts to ensure labeling correctness. In this paper, we propose DeepGD, a black-box multi-objective test selection approach for DNN models. It reduces the cost of labeling by prioritizing the selection of test inputs with high fault revealing power from large unlabeled datasets. DeepGD not only selects test inputs with high uncertainty scores to trigger as many mispredicted inputs as possible but also maximizes the probability of revealing distinct faults in the DNN model by selecting diverse mispredicted inputs. The experimental results conducted on four widely used datasets and five DNN models show that in terms of fault-revealing ability: (1) White-box, coverage-based approaches fare poorly, (2) DeepGD outperforms existing black-box test selection approaches in terms of fault detection, and (3) DeepGD also leads to better guidance for DNN model retraining when using selected inputs to augment the training set.

References in corpus (19)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Survey on Deep Learning in Medical Image Analysis
• pymoo: Multi-objective Optimization in Python
• A Survey of Deep Learning Techniques for Autonomous Driving
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• Determinantal point processes for machine learning
• Diversity in Machine Learning
• Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds
• Test Case Selection and Prioritization Using Machine Learning: A Systematic Literature Review
• Testing Deep Neural Networks
• Test Set Diameter: Quantifying the Diversity of Sets of Test Cases
• Black-Box Testing of Deep Neural Networks Through Test Case Diversity
• How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
• Boosting Operational DNN Testing Efficiency through Conditioning
• A Search-Based Testing Approach for Deep Reinforcement Learning Agents
• Simple Techniques Work Surprisingly Well for Neural Network Test Prioritization and Active Learning (Replicability Study)
• A Comparative Survey of Deep Active Learning
• Normalized Compression Distance of Multisets with Applications
• Supporting DNN Safety Analysis and Retraining through Heatmap-based Unsupervised Learning

Cited by in corpus (6)

• VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation
• LTM: Scalable and Black-box Similarity-based Test Suite Minimization based on Language Models
• SMARLA: A Safety Monitoring Approach for Deep Reinforcement Learning Agents
• A Survey of the Metrics, Uses, and Subjects of Diversity-Based Techniques in Software Testing
• TEASMA: A Practical Methodology for Test Adequacy Assessment of Deep Neural Networks
• AIM: Automated Input Set Minimization for Metamorphic Security Testing