Exposing Previously Undetectable Faults in Deep Neural Networks

Abstract

Existing methods for testing DNNs solve the oracle problem by constraining the raw features (e.g. image pixel values) to be within a small distance of a dataset example for which the desired DNN output is known. But this limits the kinds of faults these approaches are able to detect. In this paper, we introduce a novel DNN testing method that is able to find faults in DNNs that other methods cannot. The crux is that, by leveraging generative machine learning, we can generate fresh test inputs that vary in their high-level features (for images, these include object shape, location, texture, and colour). We demonstrate that our approach is capable of detecting deliberately injected faults as well as new faults in state-of-the-art DNNs, and that in both cases, existing methods are unable to find these faults.

arXiv commentAccepted to the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2021)

References in corpus (23)

• Auto-Encoding Variational Bayes
• Conditional Generative Adversarial Nets
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Shortcut Learning in Deep Neural Networks
• NIPS 2016 Tutorial: Generative Adversarial Networks
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Fast is better than free: Revisiting adversarial training
• Generating Natural Adversarial Examples
• Adversarial Examples Are Not Bugs, They Are Features
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• DLFuzz: Differential Fuzzing Testing of Deep Learning Systems
• Exploring the Landscape of Spatial Robustness
• Natural Adversarial Examples
• Motivating the Rules of the Game for Adversarial Example Research
• Constructing Unrestricted Adversarial Examples with Generative Models
• Do Adversarially Robust ImageNet Models Transfer Better?
• Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbations
• Beyond Pixel Norm-Balls: Parametric Adversaries using an Analytically Differentiable Renderer
• SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editing
• Generating Semantic Adversarial Examples via Feature Manipulation
• Evaluating Robustness to Context-Sensitive Feature Perturbations of Different Granularities
• Testing Deep Learning Models for Image Analysis Using Object-Relevant Metamorphic Relations

Cited by in corpus (2)

• Partial success in closing the gap between human and machine vision
• Targeted Deep Learning System Boundary Testing