Metamorphic Testing of a Deep Learning based Forecaster
arXiv:1907.06632
Abstract
In this paper, we present the Metamorphic Testing of an in-use deep learning based forecasting application. The application looks at the past data of system characteristics (e.g. `memory allocation') to predict outages in the future. We focus on two statistical / machine learning based components - a) detection of co-relation between system characteristics and b) estimating the future value of a system characteristic using an LSTM (a deep learning architecture). In total, 19 Metamorphic Relations have been developed and we provide proofs & algorithms where applicable. We evaluated our method through two settings. In the first, we executed the relations on the actual application and uncovered 8 issues not known before. Second, we generated hypothetical bugs, through Mutation Testing, on a reference implementation of the LSTM based forecaster and found that 65.9% of the bugs were caught through the relations.
Paper published at the 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing (MET)
References in corpus (8)
- DeepXplore: Automated Whitebox Testing of Deep Learning Systems
- Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
- DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars
- Measuring the tendency of CNNs to Learn Surface Statistical Regularities
- Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing
- Comparing deep neural networks against humans: object recognition when the signal gets weaker
- Practical Black-Box Attacks against Machine Learning
- Iterative Orthogonal Feature Projection for Diagnosing Bias in Black-Box Models