Guiding Deep Learning System Testing using Surprise Adequacy

Abstract

Deep Learning (DL) systems are rapidly being adopted in safety and security critical domains, urgently calling for ways to test their correctness and robustness. Testing of DL systems has traditionally relied on manual collection and labelling of data. Recently, a number of coverage criteria based on neuron activation values have been proposed. These criteria essentially count the number of neurons whose activation during the execution of a DL system satisfied certain properties, such as being above predefined thresholds. However, existing coverage criteria are not sufficiently fine grained to capture subtle behaviours exhibited by DL systems. Moreover, evaluations have focused on showing correlation between adversarial examples and proposed criteria rather than evaluating and guiding their use for actual testing of DL systems. We propose a novel test adequacy criterion for testing of DL systems, called Surprise Adequacy for Deep Learning Systems (SADL), which is based on the behaviour of DL systems with respect to their training data. We measure the surprise of an input as the difference in DL system's behaviour between the input and the training data (i.e., what was learnt during training), and subsequently develop this as an adequacy criterion: a good test input should be sufficiently but not overtly surprising compared to training data. Empirical evaluation using a range of DL systems from simple image classifiers to autonomous driving car platforms shows that systematic sampling of inputs based on their surprise can improve classification accuracy of DL systems against adversarial examples by up to 77.5% via retraining.

References in corpus (4)

• Sequence to Sequence Learning with Neural Networks
• Better Mixing via Deep Representations
• Combinatorial Testing for Deep Learning Systems
• DeepMutation: Mutation Testing of Deep Learning Systems

Cited by in corpus (35)

• Machine Learning Testing: Survey, Landscapes and Horizons
• Black-Box Testing of Deep Neural Networks Through Test Case Diversity
• Arachne: Search Based Repair of Deep Neural Networks
• METTLE: a METamorphic testing approach to assessing and validating unsupervised machine LEarning systems
• Comparing Offline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study
• A Survey of Safety and Trustworthiness of Deep Neural Networks: Verification, Testing, Adversarial Attack and Defence, and Interpretability
• Reducing DNN Labelling Cost using Surprise Adequacy: An Industrial Case Study for Autonomous Driving
• Coverage Guided Testing for Recurrent Neural Networks
• Supporting DNN Safety Analysis and Retraining through Heatmap-based Unsupervised Learning
• Understanding Performance Problems in Deep Learning Systems
• Testing DNN Image Classifiers for Confusion & Bias Errors
• Paracosm: A Language and Tool for Testing Autonomous Driving Systems
• Operational Calibration: Debugging Confidence Errors for DNNs in the Field
• Operation is the hardest teacher: estimating DNN accuracy looking for mispredictions
• Boundary Value Exploration for Software Analysis
• Secure Deep Learning Engineering: A Software Quality Assurance Perspective
• Neural Bug Finding: A Study of Opportunities and Challenges
• Simulator-based explanation and debugging of hazard-triggering events in DNN-based safety-critical systems
• PatchCensor: Patch Robustness Certification for Transformers via Exhaustive Testing
• Towards Security Threats of Deep Learning Systems: A Survey
• DeepGini: Prioritizing Massive Tests to Enhance the Robustness of Deep Neural Networks
• Towards Automating the AI Operations Lifecycle
• Fairness Testing of Deep Image Classification with Adequacy Metrics
• ShapeFlow: Dynamic Shape Interpreter for TensorFlow
• Revisiting Deep Neural Network Test Coverage from the Test Effectiveness Perspective
• Testing of Autonomous Driving Systems: Where Are We and Where Should We Go?
• DeepSearch: A Simple and Effective Blackbox Attack for Deep Neural Networks
• DeepFault: Fault Localization for Deep Neural Networks
• Can Offline Testing of Deep Neural Networks Replace Their Online Testing?
• Towards Structured Evaluation of Deep Neural Network Supervisors
• Explaining Image Classifiers using Statistical Fault Localization
• Structure-Invariant Testing for Machine Translation
• Testing Deep Learning Models: A First Comparative Study of Multiple Testing Techniques
• Performance Analysis of Out-of-Distribution Detection on Various Trained Neural Networks
• Testing Machine Translation via Referential Transparency