On Evaluating Adversarial Robustness

Abstract

Correctly evaluating defenses against adversarial examples has proven to be extremely difficult. Despite the significant amount of recent work attempting to design defenses that withstand adaptive attacks, few have succeeded; most papers that propose defenses are quickly shown to be incorrect. We believe a large contributing factor is the difficulty of performing security evaluations. In this paper, we discuss the methodological foundations, review commonly accepted best practices, and suggest new methods for evaluating defenses to adversarial examples. We hope that both researchers developing defenses as well as readers and reviewers who wish to understand the completeness of an evaluation consider our advice in order to avoid common pitfalls.

Living document; source available at https://github.com/evaluating-adversarial-robustness/adv-eval-paper/

References in corpus (34)

• Explaining and Harnessing Adversarial Examples
• ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Delving into Transferable Adversarial Examples and Black-box Attacks
• Certified Defenses against Adversarial Examples
• Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
• Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
• Evaluating Robustness of Neural Networks with Mixed Integer Programming
• On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models
• Defensive Distillation is Not Robust to Adversarial Examples
• Towards the first adversarially robust neural network model on MNIST
• Motivating the Rules of the Game for Adversarial Example Research
• Adversarial Examples that Fool both Computer Vision and Time-Limited Humans
• MagNet and "Efficient Defenses Against Adversarial Attacks" are Not Robust to Adversarial Examples
• Adversarial Examples Are a Natural Consequence of Test Error in Noise
• Constructing Unrestricted Adversarial Examples with Generative Models
• On the Robustness of the CVPR 2018 White-Box Adversarial Example Defenses
• Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations
• Attacking the Madry Defense Model with $L_1$-based Adversarial Examples
• Evaluating and Understanding the Robustness of Adversarial Logit Pairing
• Adversarial Example Defenses: Ensembles of Weak Defenses are not Strong
• EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examples
• Logit Pairing Methods Can Fool Gradient-Based Attacks
• AutoZOOM: Autoencoder-based Zeroth Order Optimization Method for Attacking Black-box Neural Networks
• Bypassing Feature Squeezing by Increasing Adversary Strength
• L2-Nonexpansive Neural Networks
• On the Limitation of Local Intrinsic Dimensionality for Characterizing the Subspaces of Adversarial Examples
• Comment on "Biologically inspired protection of deep networks from adversarial attacks"
• Is AmI (Attacks Meet Interpretability) Robust to Adversarial Examples?
• With Friends Like These, Who Needs Adversaries?
• On the Limitation of MagNet Defense against $L_1$-based Adversarial Examples
• The Efficacy of SHIELD under Different Threat Models

Cited by in corpus (199)

• Fast is better than free: Revisiting adversarial training
• Adversarial Examples Are Not Bugs, They Are Features
• Adversarial Machine Learning in Image Classification: A Survey Towards the Defender's Perspective
• Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning
• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
• On Adaptive Attacks to Adversarial Example Defenses
• Adversarial Attack Vulnerability of Medical Image Analysis Systems: Unexplored Factors
• Poison Ink: Robust and Invisible Backdoor Attack
• RobustBench: a standardized adversarial robustness benchmark
• FMix: Enhancing Mixed Sample Data Augmentation
• Rethinking Softmax Cross-Entropy Loss for Adversarial Robustness
• Smooth Adversarial Training
• Testing Robustness Against Unforeseen Adversaries
• Gotta Catch 'Em All: Using Honeypots to Catch Adversarial Attacks on Neural Networks
• EvadeDroid: A Practical Evasion Attack on Machine Learning for Black-box Android Malware Detection
• Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions
• Boosting Adversarial Training with Hypersphere Embedding
• Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustness
• DVERGE: Diversifying Vulnerabilities for Enhanced Robust Generation of Ensembles
• Towards Evaluating the Robustness of Deep Diagnostic Models by Adversarial Attack
• Explanation-Guided Backdoor Poisoning Attacks Against Malware Classifiers
• RobustART: Benchmarking Robustness on Architecture Design and Training Techniques
• An Alternative Surrogate Loss for PGD-based Adversarial Testing
• Adversarial Examples in Modern Machine Learning: A Review
• Adversarial Explanations for Understanding Image Classification Decisions and Improved Neural Network Robustness
• On Success and Simplicity: A Second Look at Transferable Targeted Attacks
• Backdoors in Neural Models of Source Code
• Mixup Inference: Better Exploiting Mixup to Defend Adversarial Attacks
• Learning perturbation sets for robust machine learning
• Fuzzing Automatic Differentiation in Deep-Learning Libraries
• Multi-SpacePhish: Extending the Evasion-space of Adversarial Attacks against Phishing Website Detectors using Machine Learning
• Theoretical evidence for adversarial robustness through randomization
• Adversarial Distributional Training for Robust Deep Learning
• Channel-Aware Adversarial Attacks Against Deep Learning-Based Wireless Signal Classifiers
• IF-Defense: 3D Adversarial Point Cloud Defense via Implicit Function based Restoration
• Towards a Robust and Trustworthy Machine Learning System Development: An Engineering Perspective
• Entropic Out-of-Distribution Detection: Seamless Detection of Unknown Examples
• Explainability and Adversarial Robustness for RNNs
• Learning to Defend by Learning to Attack
• A survey on Adversarial Recommender Systems: from Attack/Defense strategies to Generative Adversarial Networks
• Towards a Robust Deep Neural Network in Texts: A Survey
• The Threat of Adversarial Attacks on Machine Learning in Network Security -- A Survey
• Robust Neural Networks using Randomized Adversarial Training
• Transfer of Adversarial Robustness Between Perturbation Types
• Dos and Don'ts of Machine Learning in Computer Security
• The RFML Ecosystem: A Look at the Unique Challenges of Applying Deep Learning to Radio Frequency Applications
• Towards Understanding Fast Adversarial Training
• Adversarial Learning in Statistical Classification: A Comprehensive Review of Defenses Against Attacks
• Adversarial Machine Learning: Bayesian Perspectives
• Deep Neural Network Fingerprinting by Conferrable Adversarial Examples
• Morphence: Moving Target Defense Against Adversarial Examples
• MAB-Malware: A Reinforcement Learning Framework for Attacking Static Malware Classifiers
• Constrained Learning with Non-Convex Losses
• Benchmarking Adversarial Robustness
• Probabilistic Margins for Instance Reweighting in Adversarial Training
• When Explainability Meets Adversarial Learning: Detecting Adversarial Examples using SHAP Signatures
• Consistency Regularization for Certified Robustness of Smoothed Classifiers
• Wild Networks: Exposure of 5G Network Infrastructures to Adversarial Examples
• Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness
• What You See is Not What the Network Infers: Detecting Adversarial Examples Based on Semantic Contradiction
• On Fast Adversarial Robustness Adaptation in Model-Agnostic Meta-Learning
• Rethinking Non-idealities in Memristive Crossbars for Adversarial Robustness in Neural Networks
• Randomization matters. How to defend against strong adversarial attacks
• Exploring Adversarial Robustness of Multi-Sensor Perception Systems in Self Driving
• Deflecting Adversarial Attacks
• FenceBox: A Platform for Defeating Adversarial Examples with Data Augmentation Techniques
• Fast Minimum-norm Adversarial Attacks through Adaptive Norm Constraints
• Training Robust Deep Neural Networks via Adversarial Noise Propagation
• Biologically Inspired Mechanisms for Adversarial Robustness
• Mitigating Advanced Adversarial Attacks with More Advanced Gradient Obfuscation Techniques
• Robustifying Models Against Adversarial Attacks by Langevin Dynamics
• Are Odds Really Odd? Bypassing Statistical Detection of Adversarial Examples
• RAID: Randomized Adversarial-Input Detection for Neural Networks
• Bag of Tricks for Adversarial Training
• Adversarial Example Games
• Adversarial Image Color Transformations in Explicit Color Filter Space
• Adversarial Examples on Object Recognition: A Comprehensive Survey
• QUANOS- Adversarial Noise Sensitivity Driven Hybrid Quantization of Neural Networks
• ATHENA: A Framework based on Diverse Weak Defenses for Building Adversarial Defense
• LTD: Low Temperature Distillation for Gradient Masking-free Adversarial Training
• Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering
• Certification of embedded systems based on Machine Learning: A survey
• Evaluating the Robustness of Bayesian Neural Networks Against Different Types of Attacks
• Stateful Detection of Model Extraction Attacks
• Adversarial Pruning: A Survey and Benchmark of Pruning Methods for Adversarial Robustness
• Manifold Regularization for Locally Stable Deep Neural Networks
• WaveGuard: Understanding and Mitigating Audio Adversarial Examples
• Anomalous Example Detection in Deep Learning: A Survey
• PDA: Progressive Data Augmentation for General Robustness of Deep Neural Networks
• Adversarial Visual Robustness by Causal Intervention
• Jacobian Adversarially Regularized Networks for Robustness
• Now You See It, Now You Dont: Adversarial Vulnerabilities in Computational Pathology
• Gradient Masking and the Underestimated Robustness Threats of Differential Privacy in Deep Learning
• A critique of the DeepSec Platform for Security Analysis of Deep Learning Models
• Dynamically Sampled Nonlocal Gradients for Stronger Adversarial Attacks
• TRS: Transferability Reduced Ensemble via Encouraging Gradient Diversity and Model Smoothness
• Towards Resistant Audio Adversarial Examples
• RobFR: Benchmarking Adversarial Robustness on Face Recognition
• Softmax-based Classification is k-means Clustering: Formal Proof, Consequences for Adversarial Attacks, and Improvement through Centroid Based Tailoring
• SoK: Systematic analysis of adversarial threats against deep learning approaches for autonomous anomaly detection systems in SDN-IoT networks
• Adversarial Security Attacks and Perturbations on Machine Learning and Deep Learning Methods
• Random Noise Defense Against Query-Based Black-Box Attacks
• Developing Future Human-Centered Smart Cities: Critical Analysis of Smart City Security, Interpretability, and Ethical Challenges
• Interpreting Adversarial Examples with Attributes
• Techniques for Symbol Grounding with SATNet
• Robust Proxy: Improving Adversarial Robustness by Robust Proxy Learning
• Enhancing Certified Robustness via Smoothed Weighted Ensembling
• On Adversarial Robustness of 3D Point Cloud Classification under Adaptive Attacks
• Towards Improving Adversarial Training of NLP Models
• Towards Robust Domain Generation Algorithm Classification
• A Comprehensive Evaluation Framework for Deep Model Robustness
• Regularizers for Single-step Adversarial Training
• Stateful Detection of Black-Box Adversarial Attacks
• Rearchitecting Classification Frameworks For Increased Robustness
• Local Reweighting for Adversarial Training
• A Bayes-Optimal View on Adversarial Examples
• Towards Rapid and Robust Adversarial Training with One-Step Attacks
• Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples
• RobOT: Robustness-Oriented Testing for Deep Learning Systems
• Beware the Black-Box: on the Robustness of Recent Defenses to Adversarial Examples
• AccelAT: A Framework for Accelerating the Adversarial Training of Deep Neural Networks through Accuracy Gradient
• MagDR: Mask-guided Detection and Reconstruction for Defending Deepfakes
• HO-FMN: Hyperparameter Optimization for Fast Minimum-Norm Attacks
• Input Hessian Regularization of Neural Networks
• Adversarially Optimized Mixup for Robust Classification
• Data Quality Matters For Adversarial Training: An Empirical Study
• DeepSearch: A Simple and Effective Blackbox Attack for Deep Neural Networks
• Adversarial Item Promotion: Vulnerabilities at the Core of Top-N Recommenders that Use Images to Address Cold Start
• Improve Adversarial Robustness via Weight Penalization on Classification Layer
• Perception Improvement for Free: Exploring Imperceptible Black-box Adversarial Attacks on Image Classification
• Auditing AI models for Verified Deployment under Semantic Specifications
• A unified view on differential privacy and robustness to adversarial examples
• Adversarially Robust Learning for Security-Constrained Optimal Power Flow
• Enhanced Isotropy Maximization Loss: Seamless and High-Performance Out-of-Distribution Detection Simply Replacing the SoftMax Loss
• Audit and Assurance of AI Algorithms: A framework to ensure ethical algorithmic practices in Artificial Intelligence
• Adversarial Robustness of Stabilized NeuralODEs Might be from Obfuscated Gradients
• Adaptive Feature Alignment for Adversarial Training
• Benchmarking adversarial attacks and defenses for time-series data
• Ptolemy: Architecture Support for Robust Deep Learning
• LAFEAT: Piercing Through Adversarial Defenses with Latent Features
• ATRO: Adversarial Training with a Rejection Option
• Bandlimiting Neural Networks Against Adversarial Attacks
• On Procedural Adversarial Noise Attack And Defense
• Rethinking Image-Scaling Attacks: The Interplay Between Vulnerabilities in Machine Learning Systems
• Iterative Window Mean Filter: Thwarting Diffusion-based Adversarial Purification
• Dompteur: Taming Audio Adversarial Examples
• Understanding and Achieving Efficient Robustness with Adversarial Supervised Contrastive Learning
• On the Exploitability of Audio Machine Learning Pipelines to Surreptitious Adversarial Examples
• Random Polytope Descriptors
• Towards Deep Learning Models Resistant to Large Perturbations
• Understanding the Error in Evaluating Adversarial Robustness
• An Empirical Study on the Relation between Network Interpretability and Adversarial Robustness
• Diminishing the Effect of Adversarial Perturbations via Refining Feature Representation
• Moving Target Defense for Deep Visual Sensing against Adversarial Examples
• Building Compact and Robust Deep Neural Networks with Toeplitz Matrices
• Adversarial Semantic Collisions
• Topological Effects on Attacks Against Vertex Classification
• The Effects of Image Distribution and Task on Adversarial Robustness
• "What's in the box?!": Deflecting Adversarial Attacks by Randomly Deploying Adversarially-Disjoint Models
• Bio-inspired Robustness: A Review
• Output Randomization: A Novel Defense for both White-box and Black-box Adversarial Models
• Who is Responsible for Adversarial Defense?
• MixDefense: A Defense-in-Depth Framework for Adversarial Example Detection Based on Statistical and Semantic Analysis
• Usable Security for ML Systems in Mental Health: A Framework
• Adversarial robustness via stochastic regularization of neural activation sensitivity
• An Empirical Study of DNNs Robustification Inefficacy in Protecting Visual Recommenders
• Content-Adaptive Pixel Discretization to Improve Model Robustness
• Sampling Prediction-Matching Examples in Neural Networks: A Probabilistic Programming Approach
• Deep Repulsive Prototypes for Adversarial Robustness
• Model-Agnostic Meta-Attack: Towards Reliable Evaluation of Adversarial Robustness
• Energy Attack: On Transferring Adversarial Examples
• VirAAL: Virtual Adversarial Active Learning For NLU
• Utilizing Adversarial Targeted Attacks to Boost Adversarial Robustness
• Robust Vision-Based Cheat Detection in Competitive Gaming
• Constraining Logits by Bounded Function for Adversarial Robustness
• Sparse Coding Frontend for Robust Neural Networks
• What Can Style Transfer and Paintings Do For Model Robustness?
• Robust Assessment of Real-World Adversarial Examples
• MadNet: Using a MAD Optimization for Defending Against Adversarial Attacks
• Delving into the pixels of adversarial samples
• Tricking Adversarial Attacks To Fail
• Compressive Sensing Based Adaptive Defence Against Adversarial Images
• Luring of transferable adversarial perturbations in the black-box paradigm
• Ensemble of Models Trained by Key-based Transformed Images for Adversarially Robust Defense Against Black-box Attacks
• Combining Different V1 Brain Model Variants to Improve Robustness to Image Corruptions in CNNs
• Delving into Deep Image Prior for Adversarial Defense: A Novel Reconstruction-based Defense Framework
• Towards adversarial robustness with 01 loss neural networks
• An Empirical Review of Adversarial Defenses
• A Neuro-Inspired Autoencoding Defense Against Adversarial Perturbations
• Thwarting finite difference adversarial attacks with output randomization
• Target Training Does Adversarial Training Without Adversarial Samples
• Adversarial Attacks with Time-Scale Representations
• Attack to Fool and Explain Deep Networks
• A Closer Look at the Adversarial Robustness of Information Bottleneck Models
• Hard-label Manifolds: Unexpected Advantages of Query Efficiency for Finding On-manifold Adversarial Examples
• Evaluating adversarial robustness in simulated cerebellum
• PDPGD: Primal-Dual Proximal Gradient Descent Adversarial Attack
• Automated Decision-based Adversarial Attacks
• Multi-way Encoding for Robustness