DeepSafe: A Data-driven Approach for Checking Adversarial Robustness in Neural Networks

Abstract

Deep neural networks have become widely used, obtaining remarkable results in domains such as computer vision, speech recognition, natural language processing, audio recognition, social network filtering, machine translation, and bio-informatics, where they have produced results comparable to human experts. However, these networks can be easily fooled by adversarial perturbations: minimal changes to correctly-classified inputs, that cause the network to mis-classify them. This phenomenon represents a concern for both safety and security, but it is currently unclear how to measure a network's robustness against such perturbations. Existing techniques are limited to checking robustness around a few individual input points, providing only very limited guarantees. We propose a novel approach for automatically identifying safe regions of the input space, within which the network is robust against adversarial perturbations. The approach is data-guided, relying on clustering to identify well-defined geometric regions as candidate safe regions. We then utilize verification techniques to confirm that these regions are safe or to provide counter-examples showing that they are not safe. We also introduce the notion of targeted robustness which, for a given target label and region, ensures that a NN does not map any input in the region to the target label. We evaluated our technique on the MNIST dataset and on a neural network implementation of a controller for the next-generation Airborne Collision Avoidance System for unmanned aircraft (ACAS Xu). For these networks, our approach identified multiple regions which were completely safe as well as some which were only safe for specific labels. It also discovered several adversarial perturbations of interest.

References in corpus (3)

• Detecting Adversarial Samples from Artifacts
• Towards Proving the Adversarial Robustness of Deep Neural Networks
• On the Effectiveness of Defensive Distillation

Cited by in corpus (20)

• Adversarial Examples: Attacks and Defenses for Deep Learning
• Optimization and Abstraction: A Synergistic Approach for Analyzing Neural Network Robustness
• Software Engineering for AI-Based Systems: A Survey
• Combinatorial Testing for Deep Learning Systems
• Testing and verification of neural-network-based safety-critical control software: A systematic literature review
• Automated Verification of Neural Networks: Advances, Challenges and Perspectives
• Guidance on the Assurance of Machine Learning in Autonomous Systems (AMLAS)
• Symbolic Execution for Deep Neural Networks
• Security for Machine Learning-based Systems: Attacks and Challenges during Training and Inference
• DeepHunter: Hunting Deep Neural Network Defects via Coverage-Guided Fuzzing
• Formal Verification of Decision-Tree Ensemble Model and Detection of its Violating-input-value Ranges
• Data Sanity Check for Deep Learning Systems via Learnt Assertions
• DeepFault: Fault Localization for Deep Neural Networks
• Formal methods and software engineering for DL. Security, safety and productivity for DL systems development
• Simplifying Neural Networks using Formal Verification
• How to Learn a Model Checker
• Adversarial Ranking Attack and Defense
• Incremental Verification of Fixed-Point Implementations of Neural Networks
• Detecting Deep Neural Network Defects with Data Flow Analysis
• Using Quantifier Elimination to Enhance the Safety Assurance of Deep Neural Networks