Motivating the Rules of the Game for Adversarial Example Research

Abstract

Advances in machine learning have led to broad deployment of systems with impressive performance on important problems. Nonetheless, these systems can be induced to make errors on data that are surprisingly similar to examples the learned system handles correctly. The existence of these errors raises a variety of questions about out-of-sample generalization and whether bad actors might use such examples to abuse deployed systems. As a result of these security concerns, there has been a flurry of recent papers proposing algorithms to defend against such malicious perturbations of correctly handled examples. It is unclear how such misclassifications represent a different kind of security problem than other errors, or even other attacker-produced examples that have no specific relationship to an uncorrupted input. In this paper, we argue that adversarial example defense papers have, to date, mostly considered abstract, toy games that do not relate to any specific security concern. Furthermore, defense papers have not yet precisely described all the abilities and limitations of attackers that would be relevant in practical security. Towards this end, we establish a taxonomy of motivations, constraints, and abilities for more plausible adversaries. Finally, we provide a series of recommendations outlining a path forward for future work to more clearly articulate the threat model and perform more meaningful evaluation.

References in corpus (41)

• Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
• Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Towards Deep Neural Network Architectures Robust to Adversarial Examples
• Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks
• Certified Defenses against Adversarial Examples
• Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality
• Manifold Mixup: Better Representations by Interpolating Hidden States
• Generating Adversarial Malware Examples for Black-Box Attacks Based on GAN
• Detecting Adversarial Samples from Artifacts
• On the (Statistical) Detection of Adversarial Examples
• Robustness May Be at Odds with Accuracy
• Certifying Some Distributional Robustness with Principled Adversarial Training
• Adversarial Perturbations Against Deep Neural Networks for Malware Classification
• PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples
• A study of the effect of JPG compression on adversarial images
• Keeping the Bad Guys Out: Protecting and Vaccinating Deep Learning with JPEG Compression
• Stochastic Activation Pruning for Robust Adversarial Defense
• Measuring the tendency of CNNs to Learn Surface Statistical Regularities
• Adversarial Examples that Fool both Computer Vision and Time-Limited Humans
• Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks
• Foveation-based Mechanisms Alleviate Adversarial Examples
• Early Methods for Detecting Adversarial Images
• Deceiving End-to-End Deep Learning Malware Detectors using Adversarial Examples
• Biologically inspired protection of deep networks from adversarial attacks
• Generative Adversarial Trainer: Defense to Adversarial Perturbations with GAN
• Attacking the Madry Defense Model with $L_1$-based Adversarial Examples
• Standard detectors aren't (currently) fooled by physical adversarial stop signs
• Defending against Adversarial Images using Basis Functions Transformations
• Fortified Networks: Improving the Robustness of Deep Networks by Modeling the Manifold of Hidden Representations
• Sufficient Conditions for Idealised Models to Have No Adversarial Examples: a Theoretical and Empirical Study with Bayesian Neural Networks
• Defending Against Adversarial Attacks by Leveraging an Entire GAN
• Suppressing the Unusual: towards Robust CNNs using Symmetric Activation Functions
• Combating Adversarial Attacks Using Sparse Representations
• Robustness of Rotation-Equivariant Networks to Adversarial Perturbations
• Adversarially Robust Training through Structured Gradient Regularization
• HyperNetworks with statistical filtering for defending adversarial examples
• Featurized Bidirectional GAN: Adversarial Defense via Adversarially Learned Semantic Inference
• LatentPoison - Adversarial Attacks On The Latent Space
• Detecting Adversarial Examples via Key-based Network
• Machine vs Machine: Minimax-Optimal Defense Against Adversarial Examples

Cited by in corpus (73)

• Theoretically Principled Trade-off between Robustness and Accuracy
• On Evaluating Adversarial Robustness
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey
• A General Framework for Adversarial Examples with Objectives
• Natural Adversarial Examples
• Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning
• Adversarial Examples Are a Natural Consequence of Test Error in Noise
• Scalable agent alignment via reward modeling: a research direction
• Rademacher Complexity for Adversarially Robust Generalization
• RobustBench: a standardized adversarial robustness benchmark
• Reevaluating Adversarial Examples in Natural Language
• Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation
• Adversarial Policies: Attacking Deep Reinforcement Learning
• Unsolved Problems in ML Safety
• A Fourier Perspective on Model Robustness in Computer Vision
• Testing Robustness Against Unforeseen Adversaries
• Excessive Invariance Causes Adversarial Vulnerability
• Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbations
• Detecting and Diagnosing Adversarial Images with Class-Conditional Capsule Reconstructions
• Adversarial Machine Learning -- Industry Perspectives
• Adversarial Attacks on Time-Series Intrusion Detection for Industrial Control Systems
• Adversarial Examples - A Complete Characterisation of the Phenomenon
• Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures
• Robust Learning with Jacobian Regularization
• Pretrained Transformers Improve Out-of-Distribution Robustness
• Towards a Robust Deep Neural Network in Texts: A Survey
• Exposing Previously Undetectable Faults in Deep Neural Networks
• Adversarial Learning in Statistical Classification: A Comprehensive Review of Defenses Against Attacks
• A Survey of Black-Box Adversarial Attacks on Computer Vision Models
• Recent advances in deep learning theory
• Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness
• Improving Robustness of ML Classifiers against Realizable Evasion Attacks Using Conserved Features
• On the effectiveness of adversarial training against common corruptions
• Adversarial Color Enhancement: Generating Unrestricted Adversarial Images by Optimizing a Color Filter
• A Research Agenda: Dynamic Models to Defend Against Correlated Attacks
• Adversarial Examples on Object Recognition: A Comprehensive Survey
• Evaluating Robustness to Context-Sensitive Feature Perturbations of Different Granularities
• Security and Machine Learning in the Real World
• A Visual Analytics Framework for Adversarial Text Generation
• Improved Adversarial Robustness via Logit Regularization Methods
• Rethinking Natural Adversarial Examples for Classification Models
• Trusted Artificial Intelligence: Towards Certification of Machine Learning Applications
• Improving VAEs' Robustness to Adversarial Attack
• Feature-level Malware Obfuscation in Deep Learning
• The Efficacy of SHIELD under Different Threat Models
• Regional Adversarial Training for Better Robust Generalization
• On the relationship between class selectivity, dimensionality, and robustness
• Identifying Model Weakness with Adversarial Examiner
• Calibrated neighborhood aware confidence measure for deep metric learning
• Talk Proposal: Towards the Realistic Evaluation of Evasion Attacks using CARLA
• Robustness, Privacy, and Generalization of Adversarial Training
• Adversarial Examples for Cost-Sensitive Classifiers
• Dompteur: Taming Audio Adversarial Examples
• On the Exploitability of Audio Machine Learning Pipelines to Surreptitious Adversarial Examples
• Towards Identifying and closing Gaps in Assurance of autonomous Road vehicleS -- a collection of Technical Notes Part 1
• Generating Structured Adversarial Attacks Using Frank-Wolfe Method
• Knowing When to Stop: Evaluation and Verification of Conformity to Output-size Specifications
• Deep Latent Defence
• Empirically Measuring Concentration: Fundamental Limits on Intrinsic Robustness
• Natural Adversarial Objects
• Domain Invariant Adversarial Learning
• Lower Bounds on Cross-Entropy Loss in the Presence of Test-time Adversaries
• Downscaling Attack and Defense: Turning What You See Back Into What You Get
• A Multiclass Boosting Framework for Achieving Fast and Provable Adversarial Robustness
• Adversarial Examples as an Input-Fault Tolerance Problem
• Deep Detector Health Management under Adversarial Campaigns
• On Universalized Adversarial and Invariant Perturbations
• Linking average- and worst-case perturbation robustness via class selectivity and dimensionality
• Trace-Norm Adversarial Examples
• When Should You Defend Your Classifier -- A Game-theoretical Analysis of Countermeasures against Adversarial Examples
• Law and Adversarial Machine Learning
• Mental Models of Adversarial Machine Learning