Guiding Deep Learning System Testing using Surprise Adequacy

Abstract

Deep Learning (DL) systems are rapidly being adopted in safety and security critical domains, urgently calling for ways to test their correctness and robustness. Testing of DL systems has traditionally relied on manual collection and labelling of data. Recently, a number of coverage criteria based on neuron activation values have been proposed. These criteria essentially count the number of neurons whose activation during the execution of a DL system satisfied certain properties, such as being above predefined thresholds. However, existing coverage criteria are not sufficiently fine grained to capture subtle behaviours exhibited by DL systems. Moreover, evaluations have focused on showing correlation between adversarial examples and proposed criteria rather than evaluating and guiding their use for actual testing of DL systems. We propose a novel test adequacy criterion for testing of DL systems, called Surprise Adequacy for Deep Learning Systems (SADL), which is based on the behaviour of DL systems with respect to their training data. We measure the surprise of an input as the difference in DL system's behaviour between the input and the training data (i.e., what was learnt during training), and subsequently develop this as an adequacy criterion: a good test input should be sufficiently but not overtly surprising compared to training data. Empirical evaluation using a range of DL systems from simple image classifiers to autonomous driving car platforms shows that systematic sampling of inputs based on their surprise can improve classification accuracy of DL systems against adversarial examples by up to 77.5% via retraining.

References in corpus (10)

• Sequence to Sequence Learning with Neural Networks
• End to End Learning for Self-Driving Cars
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality
• Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
• Better Mixing via Deep Representations
• DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
• Test Set Diameter: Quantifying the Diversity of Sets of Test Cases
• Combinatorial Testing for Deep Learning Systems
• DeepMutation: Mutation Testing of Deep Learning Systems

Cited by in corpus (36)

• Testing Deep Neural Networks
• Machine Learning Testing: Survey, Landscapes and Horizons
• Black-Box Testing of Deep Neural Networks Through Test Case Diversity
• Arachne: Search Based Repair of Deep Neural Networks
• METTLE: a METamorphic testing approach to assessing and validating unsupervised machine LEarning systems
• Comparing Offline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study
• A Survey of Safety and Trustworthiness of Deep Neural Networks: Verification, Testing, Adversarial Attack and Defence, and Interpretability
• Reducing DNN Labelling Cost using Surprise Adequacy: An Industrial Case Study for Autonomous Driving
• Coverage Guided Testing for Recurrent Neural Networks
• Supporting DNN Safety Analysis and Retraining through Heatmap-based Unsupervised Learning
• Understanding Performance Problems in Deep Learning Systems
• Testing DNN Image Classifiers for Confusion & Bias Errors
• Paracosm: A Language and Tool for Testing Autonomous Driving Systems
• Operational Calibration: Debugging Confidence Errors for DNNs in the Field
• Operation is the hardest teacher: estimating DNN accuracy looking for mispredictions
• Secure Deep Learning Engineering: A Software Quality Assurance Perspective
• Boundary Value Exploration for Software Analysis
• Simulator-based explanation and debugging of hazard-triggering events in DNN-based safety-critical systems
• Neural Bug Finding: A Study of Opportunities and Challenges
• PatchCensor: Patch Robustness Certification for Transformers via Exhaustive Testing
• Towards Security Threats of Deep Learning Systems: A Survey
• DeepGini: Prioritizing Massive Tests to Enhance the Robustness of Deep Neural Networks
• Towards Automating the AI Operations Lifecycle
• ShapeFlow: Dynamic Shape Interpreter for TensorFlow
• Fairness Testing of Deep Image Classification with Adequacy Metrics
• Revisiting Deep Neural Network Test Coverage from the Test Effectiveness Perspective
• DeepSearch: A Simple and Effective Blackbox Attack for Deep Neural Networks
• Testing of Autonomous Driving Systems: Where Are We and Where Should We Go?
• Can Offline Testing of Deep Neural Networks Replace Their Online Testing?
• DeepFault: Fault Localization for Deep Neural Networks
• Testing Deep Learning Models: A First Comparative Study of Multiple Testing Techniques
• Structure-Invariant Testing for Machine Translation
• Towards Structured Evaluation of Deep Neural Network Supervisors
• Explaining Image Classifiers using Statistical Fault Localization
• Testing Machine Translation via Referential Transparency
• Performance Analysis of Out-of-Distribution Detection on Various Trained Neural Networks