Coverage Testing of Deep Learning Models using Dataset Characterization
arXiv:1911.07309
Abstract
Deep Neural Networks (DNNs), with its promising performance, are being increasingly used in safety critical applications such as autonomous driving, cancer detection, and secure authentication. With growing importance in deep learning, there is a requirement for a more standardized framework to evaluate and test deep learning models. The primary challenge involved in automated generation of extensive test cases are: (i) neural networks are difficult to interpret and debug and (ii) availability of human annotators to generate specialized test points. In this research, we explain the necessity to measure the quality of a dataset and propose a test case generation system guided by the dataset properties. From a testing perspective, four different dataset quality dimensions are proposed: (i) equivalence partitioning, (ii) centroid positioning, (iii) boundary conditioning, and (iv) pair-wise boundary conditioning. The proposed system is evaluated on well known image classification datasets such as MNIST, Fashion-MNIST, CIFAR10, CIFAR100, and SVHN against popular deep learning models such as LeNet, ResNet-20, VGG-19. Further, we conduct various experiments to demonstrate the effectiveness of systematic test case generation system for evaluating deep learning models.
References in corpus (8)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- End to End Learning for Self-Driving Cars
- DeepXplore: Automated Whitebox Testing of Deep Learning Systems
- An Empirical Evaluation of Deep Learning on Highway Driving
- Testing Deep Neural Networks
- TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing
- DeepMutation: Mutation Testing of Deep Learning Systems
Cited by in corpus (4)
- How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
- On the Impact of Data Quality on Image Classification Fairness
- Assessing and Accelerating Coverage in Deep Reinforcement Learning
- A Study of Domain Generalization on Ultrasound-based Multi-Class Segmentation of Arteries, Veins, Ligaments, and Nerves Using Transfer Learning