Towards Practical Verification of Machine Learning: The Case of Computer Vision Systems

Abstract

Due to the increasing usage of machine learning (ML) techniques in security- and safety-critical domains, such as autonomous systems and medical diagnosis, ensuring correct behavior of ML systems, especially for different corner cases, is of growing importance. In this paper, we propose a generic framework for evaluating security and robustness of ML systems using different real-world safety properties. We further design, implement and evaluate VeriVis, a scalable methodology that can verify a diverse set of safety properties for state-of-the-art computer vision systems with only blackbox access. VeriVis leverage different input space reduction techniques for efficient verification of different safety properties. VeriVis is able to find thousands of safety violations in fifteen state-of-the-art computer vision systems including ten Deep Neural Networks (DNNs) such as Inception-v3 and Nvidia's Dave self-driving system with thousands of neurons as well as five commercial third-party vision APIs including Google vision and Clarifai for twelve different safety properties. Furthermore, VeriVis can successfully verify local safety properties, on average, for around 31.7% of the test images. VeriVis finds up to 64.8x more violations than existing gradient-based methods that, unlike VeriVis, cannot ensure non-existence of any violations. Finally, we show that retraining using the safety violations detected by VeriVis can reduce the average number of violations up to 60.2%.

References in corpus (11)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• End to End Learning for Self-Driving Cars
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• Delving into Transferable Adversarial Examples and Black-box Attacks
• Towards Deep Neural Network Architectures Robust to Adversarial Examples
• Xception: Deep Learning with Depthwise Separable Convolutions
• DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars
• Parseval Networks: Improving Robustness to Adversarial Examples
• Improving the Robustness of Deep Neural Networks via Stability Training
• Can we still avoid automatic face detection?
• Toward Guaranteed Illumination Models for Non-Convex Objects

Cited by in corpus (20)

• Formal Security Analysis of Neural Networks using Symbolic Intervals
• Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
• How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
• Optimization and Abstraction: A Synergistic Approach for Analyzing Neural Network Robustness
• Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions
• Combinatorial Testing for Deep Learning Systems
• A Survey of Safety and Trustworthiness of Deep Neural Networks: Verification, Testing, Adversarial Attack and Defence, and Interpretability
• DeepHunter: Hunting Deep Neural Network Defects via Coverage-Guided Fuzzing
• Formal Verification of CNN-based Perception Systems
• Adversarial Examples on Object Recognition: A Comprehensive Survey
• What Does CNN Shift Invariance Look Like? A Visualization Study
• Adversarial Attacks Beyond the Image Space
• Invariance-inducing regularization using worst-case transformations suffices to boost accuracy and spatial robustness
• TSS: Transformation-Specific Smoothing for Robustness Certification
• Identifying Model Weakness with Adversarial Examiner
• Risk Assessment for Machine Learning Models
• Automated Testing for Deep Learning Systems with Differential Behavior Criteria
• Formal Verification of End-to-End Learning in Cyber-Physical Systems: Progress and Challenges
• Accelerating Entrepreneurial Decision-Making Through Hybrid Intelligence
• The future of human-AI collaboration: a taxonomy of design knowledge for hybrid intelligence systems