DeepSmartFuzzer: Reward Guided Test Generation For Deep Learning
arXiv:1911.10621
Abstract
Testing Deep Neural Network (DNN) models has become more important than ever with the increasing usage of DNN models in safety-critical domains such as autonomous cars. The traditional approach of testing DNNs is to create a test set, which is a random subset of the dataset about the problem of interest. This kind of approach is not enough for testing most of the real-world scenarios since these traditional test sets do not include corner cases, while a corner case input is generally considered to introduce erroneous behaviors. Recent works on adversarial input generation, data augmentation, and coverage-guided fuzzing (CGF) have provided new ways to extend traditional test sets. Among those, CGF aims to produce new test inputs by fuzzing existing ones to achieve high coverage on a test adequacy criterion (i.e. coverage criterion). Given that the subject test adequacy criterion is a well-established one, CGF can potentially find error inducing inputs for different underlying reasons. In this paper, we propose a novel CGF solution for structural testing of DNNs. The proposed fuzzer employs Monte Carlo Tree Search to drive the coverage-guided search in the pursuit of achieving high coverage. Our evaluation shows that the inputs generated by our method result in higher coverage than the inputs produced by the previously introduced coverage-guided fuzzing techniques.
References in corpus (13)
- A Survey on Deep Learning in Medical Image Analysis
- Sequence to Sequence Learning with Neural Networks
- End to End Learning for Self-Driving Cars
- Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
- DeepXplore: Automated Whitebox Testing of Deep Learning Systems
- DLFuzz: Differential Fuzzing Testing of Deep Learning Systems
- Testing Deep Neural Networks
- TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing
- Machine Learning Testing: Survey, Landscapes and Horizons
- Learning to Explain: An Information-Theoretic Perspective on Model Interpretation
- DeepRoad: GAN-based Metamorphic Autonomous Driving System Testing
- Symbolic Execution for Deep Neural Networks
- DeepHunter: Hunting Deep Neural Network Defects via Coverage-Guided Fuzzing