A Survey of Safety and Trustworthiness of Deep Neural Networks: Verification, Testing, Adversarial Attack and Defence, and Interpretability

Abstract

In the past few years, significant progress has been made on deep neural networks (DNNs) in achieving human-level performance on several long-standing tasks. With the broader deployment of DNNs on various applications, the concerns over their safety and trustworthiness have been raised in public, especially after the widely reported fatal incidents involving self-driving cars. Research to address these concerns is particularly active, with a significant number of papers released in the past few years. This survey paper conducts a review of the current research effort into making DNNs safe and trustworthy, by focusing on four aspects: verification, testing, adversarial attack and defence, and interpretability. In total, we survey 202 papers, most of which were published after 2017.

arXiv commentTo appear in the journal of Computer Science Review

References in corpus (52)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Explaining and Harnessing Adversarial Examples
• Natural Language Processing (almost) from Scratch
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Axiomatic Attribution for Deep Networks
• Striving for Simplicity: The All Convolutional Net
• Learning Important Features Through Propagating Activation Differences
• Ensemble Adversarial Training: Attacks and Defenses
• Prioritized Experience Replay
• Conditional Image Generation with PixelCNN Decoders
• Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
• Dueling Network Architectures for Deep Reinforcement Learning
• Understanding Neural Networks Through Deep Visualization
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• Opening the Black Box of Deep Neural Networks via Information
• SmoothGrad: removing noise by adding noise
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Certified Defenses against Adversarial Examples
• Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality
• Guiding Deep Learning System Testing using Surprise Adequacy
• Countering Adversarial Images using Input Transformations
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
• Efficient Neural Network Robustness Certification with General Activation Functions
• DLFuzz: Differential Fuzzing Testing of Deep Learning Systems
• Formal Security Analysis of Neural Networks using Symbolic Intervals
• Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation
• An approach to reachability analysis for feed-forward ReLU neural networks
• NO Need to Worry about Adversarial Examples in Object Detection in Autonomous Vehicles
• Automated Directed Fairness Testing
• Testing Deep Neural Networks
• MagNet and "Efficient Defenses Against Adversarial Attacks" are Not Robust to Adversarial Examples
• TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing
• Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamples
• Early Methods for Detecting Adversarial Images
• A Game-Based Approximate Verification of Deep Neural Networks with Provable Guarantees
• Adversarial Attacks Against Medical Deep Learning Systems
• Towards Practical Verification of Machine Learning: The Case of Computer Vision Systems
• DeepRoad: GAN-based Metamorphic Autonomous Driving System Testing
• Combinatorial Testing for Deep Learning Systems
• Symbolic Execution for Deep Neural Networks
• Using Machine Learning Safely in Automotive Software: An Assessment and Adaption of Software Process Requirements in ISO 26262
• Systematic Testing of Convolutional Neural Networks for Autonomous Driving
• DeepHunter: Hunting Deep Neural Network Defects via Coverage-Guided Fuzzing
• Automated Test Generation to Detect Individual Discrimination in AI Models
• Global Robustness Evaluation of Deep Neural Networks with Provable Guarantees for the $L_0$ Norm
• Gradient Masking Causes CLEVER to Overestimate Adversarial Perturbation Size
• Sequential Attacks on Agents for Long-Term Adversarial Goals
• Sparse Adversarial Perturbations for Videos
• Runtime Monitoring Neuron Activation Patterns
• nn-dependability-kit: Engineering Neural Networks for Safety-Critical Autonomous Driving Systems

Cited by in corpus (17)

• Testing Deep Neural Networks
• Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges
• Detecting Adversarial Examples by Input Transformations, Defense Perturbations, and Voting
• Multi-Label Classification Neural Networks with Hard Logical Constraints
• The RFML Ecosystem: A Look at the Unique Challenges of Applying Deep Learning to Radio Frequency Applications
• Yes We Care! -- Certification for Machine Learning Methods through the Care Label Framework
• Adversarial Examples on Object Recognition: A Comprehensive Survey
• Trust but Verify: An Information-Theoretic Explanation for the Adversarial Fragility of Machine Learning Systems, and a General Defense against Adversarial Attacks
• Architecting Dependable Learning-enabled Autonomous Systems: A Survey
• Reliability Validation of Learning Enabled Vehicle Tracking
• Towards Interpreting Recurrent Neural Networks through Probabilistic Abstraction
• An Adaptive View of Adversarial Robustness from Test-time Smoothing Defense
• Detecting Out-of-distribution Objects Using Neuron Activation Patterns
• Generating Adversarial Inputs Using A Black-box Differential Technique
• Towards Safety Verification of Direct Perception Neural Networks
• NeVer 2.0: Learning, Verification and Repair of Deep Neural Networks
• Using Quantifier Elimination to Enhance the Safety Assurance of Deep Neural Networks