Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing

Abstract

We have recently witnessed tremendous success of Machine Learning (ML) in practical applications. Computer vision, speech recognition and language translation have all seen a near human level performance. We expect, in the near future, most business applications will have some form of ML. However, testing such applications is extremely challenging and would be very expensive if we follow today's methodologies. In this work, we present an articulation of the challenges in testing ML based applications. We then present our solution approach, based on the concept of Metamorphic Testing, which aims to identify implementation bugs in ML based image classifiers. We have developed metamorphic relations for an application based on Support Vector Machine and a Deep Learning based application. Empirical validation showed that our approach was able to catch 71% of the implementation bugs in the ML applications.

Published at 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2018)

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• The Loss Surfaces of Multilayer Networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars
• Qualitatively characterizing neural network optimization problems
• Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing
• Practical Black-Box Attacks against Machine Learning

Cited by in corpus (33)

• Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing
• Testing Deep Neural Networks
• Machine Learning Testing: Survey, Landscapes and Horizons
• Machine Learning for Software Engineering: A Systematic Mapping
• METTLE: a METamorphic testing approach to assessing and validating unsupervised machine LEarning systems
• Software Engineering for AI-Based Systems: A Survey
• Quality Assurance for AI-based Systems: Overview and Challenges
• Software Testing for Machine Learning
• A Survey on Verification and Validation, Testing and Evaluations of Neurosymbolic Artificial Intelligence
• Perfect is the enemy of test oracle
• GenMorph: Automatically Generating Metamorphic Relations via Genetic Programming
• If a Human Can See It, So Should Your System: Reliability Requirements for Machine Vision Components
• Metamorphic Testing for Object Detection Systems
• Synergy between Machine/Deep Learning and Software Engineering: How Far Are We?
• Metamorphic Detection of Adversarial Examples in Deep Learning Models With Affine Transformations
• Towards Testing of Deep Learning Systems with Training Set Reduction
• A Systematic Mapping Study on Testing of Machine Learning Programs
• Semantic-Based Neural Network Repair
• ShapeFlow: Dynamic Shape Interpreter for TensorFlow
• 2L-3W: 2-Level 3-Way Hardware-Software Co-Verification for the Mapping of Deep Learning Architecture (DLA) onto FPGA Boards
• Opening the Software Engineering Toolbox for the Assessment of Trustworthy AI
• ExAIS: Executable AI Semantics
• Testing Deep Learning Models for Image Analysis Using Object-Relevant Metamorphic Relations
• Metamorphic Testing for Pose Estimation Systems
• Metamorphic Testing of Image Captioning Systems via Image-Level Reduction
• Testing of Autonomous Driving Systems: Where Are We and Where Should We Go?
• Testing Untestable Neural Machine Translation: An Industrial Case
• Object-based Metamorphic Testing through Image Structuring
• Understanding the Nature of System-Related Issues in Machine Learning Frameworks: An Exploratory Study
• Testing Deep Learning Models: A First Comparative Study of Multiple Testing Techniques
• Reliability Validation of Learning Enabled Vehicle Tracking
• Metamorphic Testing of a Deep Learning based Forecaster
• Detecting Deep Neural Network Defects with Data Flow Analysis