Simple Techniques Work Surprisingly Well for Neural Network Test Prioritization and Active Learning (Replicability Study)

Abstract

Test Input Prioritizers (TIP) for Deep Neural Networks (DNN) are an important technique to handle the typically very large test datasets efficiently, saving computation and labeling costs. This is particularly true for large-scale, deployed systems, where inputs observed in production are recorded to serve as potential test or training data for the next versions of the system. Feng et. al. propose DeepGini, a very fast and simple TIP, and show that it outperforms more elaborate techniques such as neuron- and surprise coverage. In a large-scale study (4 case studies, 8 test datasets, 32'200 trained models) we verify their findings. However, we also find that other comparable or even simpler baselines from the field of uncertainty quantification, such as the predicted softmax likelihood or the entropy of the predicted softmax likelihoods perform equally well as DeepGini.

arXiv commentAccepted at ISSTA 2022

References in corpus (2)

• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems

Cited by in corpus (6)

• Mind the Gap! A Study on the Transferability of Virtual vs Physical-world Testing of Autonomous Driving Systems
• DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks
• Dynamic Test Case Prioritization in Industrial Test Result Datasets
• CheapET-3: Cost-Efficient Use of Remote DNN Models
• Toward Efficient Testing of Graph Neural Networks via Test Input Prioritization
• To Softmax, or not to Softmax: that is the question when applying Active Learning for Transformer Models