DeepHunter: Hunting Deep Neural Network Defects via Coverage-Guided Fuzzing

Abstract

In company with the data explosion over the past decade, deep neural network (DNN) based software has experienced unprecedented leap and is becoming the key driving force of many novel industrial applications, including many safety-critical scenarios such as autonomous driving. Despite great success achieved in various human intelligence tasks, similar to traditional software, DNNs could also exhibit incorrect behaviors caused by hidden defects causing severe accidents and losses. In this paper, we propose DeepHunter, an automated fuzz testing framework for hunting potential defects of general-purpose DNNs. DeepHunter performs metamorphic mutation to generate new semantically preserved tests, and leverages multiple plugable coverage criteria as feedback to guide the test generation from different perspectives. To be scalable towards practical-sized DNNs, DeepHunter maintains multiple tests in a batch, and prioritizes the tests selection based on active feedback. The effectiveness of DeepHunter is extensively investigated on 3 popular datasets (MNIST, CIFAR-10, ImageNet) and 7 DNNs with diverse complexities, under a large set of 6 coverage criteria as feedback. The large-scale experiments demonstrate that DeepHunter can (1) significantly boost the coverage with guidance; (2) generate useful tests to detect erroneous behaviors and facilitate the DNN model quality evaluation; (3) accurately capture potential defects during DNN quantization for platform migration.

References in corpus (18)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Ensemble Adversarial Training: Attacks and Defenses
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• Generating Natural Adversarial Examples
• Training and Inference with Integers in Deep Neural Networks
• PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples
• Spatially Transformed Adversarial Examples
• Towards Vision-Based Deep Reinforcement Learning for Robotic Motion Control
• Stochastic Activation Pruning for Robust Adversarial Defense
• TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing
• Not all bytes are equal: Neural byte sieve for fuzzing
• Towards Practical Verification of Machine Learning: The Case of Computer Vision Systems
• DeepSafe: A Data-driven Approach for Checking Adversarial Robustness in Neural Networks
• Combinatorial Testing for Deep Learning Systems
• DeepMutation: Mutation Testing of Deep Learning Systems
• Attacking Binarized Neural Networks

Cited by in corpus (8)

• Testing Deep Neural Networks
• A Survey of Safety and Trustworthiness of Deep Neural Networks: Verification, Testing, Adversarial Attack and Defence, and Interpretability
• Metamorphic Testing for Object Detection Systems
• Test Selection for Deep Learning Systems
• CAGFuzz: Coverage-Guided Adversarial Generative Fuzzing Testing of Deep Learning Systems
• DeepSmartFuzzer: Reward Guided Test Generation For Deep Learning
• Reliability Validation of Learning Enabled Vehicle Tracking
• Graph-Based Fuzz Testing for Deep Learning Inference Engine