HOMRS: High Order Metamorphic Relations Selector for Deep Neural Networks
arXiv:2107.04863
Abstract
Deep Neural Networks (DNN) applications are increasingly becoming a part of our everyday life, from medical applications to autonomous cars. Traditional validation of DNN relies on accuracy measures, however, the existence of adversarial examples has highlighted the limitations of these accuracy measures, raising concerns especially when DNN are integrated into safety-critical systems. In this paper, we present HOMRS, an approach to boost metamorphic testing by automatically building a small optimized set of high order metamorphic relations from an initial set of elementary metamorphic relations. HOMRS' backbone is a multi-objective search; it exploits ideas drawn from traditional systems testing such as code coverage, test case, path diversity as well as input validation. We applied HOMRS to MNIST/LeNet and SVHN/VGG and we report evidence that it builds a small but effective set of high-order transformations that generalize well to the input data distribution. Moreover, comparing to similar generation technique such as DeepXplore, we show that our distribution-based approach is more effective, generating valid transformations from an uncertainty quantification point of view, while requiring less computation time by leveraging the generalization ability of the approach.
33 pages
References in corpus (5)
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning
- Metamorphic Testing: A New Approach for Generating Next Test Cases
- Metamorphic Detection of Adversarial Examples in Deep Learning Models With Affine Transformations
- Metamorphic Testing of a Deep Learning based Forecaster