Adversarial Risk and the Dangers of Evaluating Against Weak Attacks

Abstract

This paper investigates recently proposed approaches for defending against adversarial examples and evaluating adversarial robustness. We motivate 'adversarial risk' as an objective for achieving models robust to worst-case inputs. We then frame commonly used attacks and evaluation metrics as defining a tractable surrogate objective to the true adversarial risk. This suggests that models may optimize this surrogate rather than the true adversarial risk. We formalize this notion as 'obscurity to an adversary,' and develop tools and heuristics for identifying obscured models and designing transparent models. We demonstrate that this is a significant problem in practice by repurposing gradient-free optimization techniques into adversarial attacks, which we use to decrease the accuracy of several recently proposed defenses to near zero. Our hope is that our formulations and results will help researchers to develop more powerful defenses.

References in corpus (23)

• Adam: A Method for Stochastic Optimization
• Ensemble Adversarial Training: Attacks and Defenses
• ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models
• Pixel Recurrent Neural Networks
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Towards Deep Neural Network Architectures Robust to Adversarial Examples
• Countering Adversarial Images using Input Transformations
• Certifying Some Distributional Robustness with Principled Adversarial Training
• Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
• Adversarial Perturbations Against Deep Neural Networks for Malware Classification
• PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples
• Adversarial Examples for Evaluating Reading Comprehension Systems
• Adversarial Examples: Attacks and Defenses for Deep Learning
• MagNet: a Two-Pronged Defense against Adversarial Examples
• Mitigating Adversarial Effects Through Randomization
• Provably Minimally-Distorted Adversarial Examples
• PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications
• Increasing the Interpretability of Recurrent Neural Networks Using Hidden Markov Models
• Query-Efficient Black-box Adversarial Examples (superceded)
• Defense against Adversarial Attacks Using High-Level Representation Guided Denoiser

Cited by in corpus (156)

• On Evaluating Adversarial Robustness
• Fast is better than free: Revisiting adversarial training
• Adversarial Examples Are Not Bugs, They Are Features
• Robustness May Be at Odds with Accuracy
• Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty
• On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models
• Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers
• Adversarial Weight Perturbation Helps Robust Generalization
• Are Labels Required for Improving Adversarial Robustness?
• Improving Black-box Adversarial Attacks with a Transfer-based Prior
• A Dual Approach to Scalable Verification of Deep Networks
• Adversarial Machine Learning in Image Classification: A Survey Towards the Defender's Perspective
• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
• On Adaptive Attacks to Adversarial Example Defenses
• Adversarial Attack Vulnerability of Medical Image Analysis Systems: Unexplored Factors
• Scalable agent alignment via reward modeling: a research direction
• On the Robustness of the CVPR 2018 White-Box Adversarial Example Defenses
• Interpreting and Improving Adversarial Robustness of Deep Neural Networks with Neuron Sensitivity
• Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability
• Fixing Data Augmentation to Improve Adversarial Robustness
• Rethinking Softmax Cross-Entropy Loss for Adversarial Robustness
• MMA Training: Direct Input Space Margin Maximization through Adversarial Training
• Adversarial Policies: Attacking Deep Reinforcement Learning
• Training verified learners with learned verifiers
• Functional Adversarial Attacks
• Evaluating and Understanding the Robustness of Adversarial Logit Pairing
• Gotta Catch 'Em All: Using Honeypots to Catch Adversarial Attacks on Neural Networks
• Accurate, reliable and fast robustness evaluation
• Boosting Adversarial Training with Hypersphere Embedding
• Security and Privacy Issues in Deep Learning
• ME-Net: Towards Effective Adversarial Robustness with Matrix Estimation
• Robust Local Features for Improving the Generalization of Adversarial Training
• RobustART: Benchmarking Robustness on Architecture Design and Training Techniques
• An Alternative Surrogate Loss for PGD-based Adversarial Testing
• Towards robust sensing for Autonomous Vehicles: An adversarial perspective
• Certified Adversarial Robustness for Deep Reinforcement Learning
• Diversity can be Transferred: Output Diversification for White- and Black-box Attacks
• Entangled Watermarks as a Defense against Model Extraction
• Logit Pairing Methods Can Fool Gradient-Based Attacks
• Defending Against Universal Attacks Through Selective Feature Regeneration
• Robustness Verification of Tree-based Models
• Saccader: Improving Accuracy of Hard Attention Models for Vision
• Curse of Dimensionality on Randomized Smoothing for Certifiable Robustness
• Denoised Smoothing: A Provable Defense for Pretrained Classifiers
• Re-evaluating Evaluation
• Adversarial Distributional Training for Robust Deep Learning
• Perturbing Across the Feature Hierarchy to Improve Standard and Strict Blackbox Attack Transferability
• Randomized Smoothing of All Shapes and Sizes
• Are Generative Classifiers More Robust to Adversarial Attacks?
• The Threat of Adversarial Attacks on Machine Learning in Network Security -- A Survey
• Harnessing the Vulnerability of Latent Layers in Adversarially Trained Models
• Morphence: Moving Target Defense Against Adversarial Examples
• Adversarial Neural Pruning with Latent Vulnerability Suppression
• Consistency Regularization for Certified Robustness of Smoothed Classifiers
• Deep Nets: What have they ever done for Vision?
• Feature Distillation: DNN-Oriented JPEG Compression Against Adversarial Examples
• Evolving Robust Neural Architectures to Defend from Adversarial Attacks
• Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation
• InfoAT: Improving Adversarial Training Using the Information Bottleneck Principle
• Adversarial Training Versus Weight Decay
• Breaking certified defenses: Semantic adversarial examples with spoofed robustness certificates
• Certifiably Robust Interpretation in Deep Learning
• A Real-time Defense against Website Fingerprinting Attacks
• Enabling certification of verification-agnostic networks via memory-efficient semidefinite programming
• Fast Minimum-norm Adversarial Attacks through Adaptive Norm Constraints
• Physical Adversarial Attacks on Deep Neural Networks for Traffic Sign Recognition: A Feasibility Study
• Robustifying Models Against Adversarial Attacks by Langevin Dynamics
• Are Odds Really Odd? Bypassing Statistical Detection of Adversarial Examples
• PatchAttack: A Black-box Texture-based Attack with Reinforcement Learning
• When Attackers Meet AI: Learning-empowered Attacks in Cooperative Spectrum Sensing
• Bag of Tricks for Adversarial Training
• LTD: Low Temperature Distillation for Gradient Masking-free Adversarial Training
• Post-breach Recovery: Protection against White-box Adversarial Examples for Leaked DNN Models
• Heuristic Black-box Adversarial Attacks on Video Recognition Models
• Meta-Learning surrogate models for sequential decision making
• Adversarial Robustness Assessment: Why both $L_0$ and $L_{\infty }$ Attacks Are Necessary
• Certifying Confidence via Randomized Smoothing
• Fine-grained Synthesis of Unrestricted Adversarial Examples
• Enhancing Cross-task Black-Box Transferability of Adversarial Examples with Dispersion Reduction
• Adversarial Visual Robustness by Causal Intervention
• Jacobian Adversarially Regularized Networks for Robustness
• Lagrangian Decomposition for Neural Network Verification
• Increasing-Margin Adversarial (IMA) Training to Improve Adversarial Robustness of Neural Networks
• Imbalanced Gradients: A Subtle Cause of Overestimated Adversarial Robustness
• Improved Adversarial Robustness via Logit Regularization Methods
• Better the Devil you Know: An Analysis of Evasion Attacks using Out-of-Distribution Adversarial Examples
• Second-Order Provable Defenses against Adversarial Attacks
• LiBRe: A Practical Bayesian Approach to Adversarial Detection
• Dynamically Sampled Nonlocal Gradients for Stronger Adversarial Attacks
• Detecting Overfitting via Adversarial Examples
• Detecting Adversarial Examples via Neural Fingerprinting
• Improving the Tightness of Convex Relaxation Bounds for Training Certifiably Robust Classifiers
• CAP-GAN: Towards Adversarial Robustness with Cycle-consistent Attentional Purification
• Adversarial Diffusion Attacks on Graph-based Traffic Prediction Models
• Second Order Optimization for Adversarial Robustness and Interpretability
• Strength in Numbers: Trading-off Robustness and Computation via Adversarially-Trained Ensembles
• Boosting the Certified Robustness of L-infinity Distance Nets
• On Adversarial Robustness of 3D Point Cloud Classification under Adaptive Attacks
• Towards Rapid and Robust Adversarial Training with One-Step Attacks
• Stateful Detection of Black-Box Adversarial Attacks
• A Method for Computing Class-wise Universal Adversarial Perturbations
• Almost Tight L0-norm Certified Robustness of Top-k Predictions against Adversarial Perturbations
• Data-Dependent Randomized Smoothing
• Towards Defending Multiple ${\ell }_p$-norm Bounded Adversarial Perturbations via Gated Batch Normalization
• A Comprehensive Evaluation Framework for Deep Model Robustness
• Robustness Certificates Against Adversarial Examples for ReLU Networks
• Enhancing Certified Robustness via Smoothed Weighted Ensembling
• Spanning Attack: Reinforce Black-box Attacks with Unlabeled Data
• Meta-Learning with Neural Tangent Kernels
• DeepSearch: A Simple and Effective Blackbox Attack for Deep Neural Networks
• Improving Robustness using Generated Data
• Improve robustness of DNN for ECG signal classification:a noise-to-signal ratio perspective
• Data Quality Matters For Adversarial Training: An Empirical Study
• Adversarial Attack across Datasets
• Certifiable Robustness to Adversarial State Uncertainty in Deep Reinforcement Learning
• Adversarial Robustness of Stabilized NeuralODEs Might be from Obfuscated Gradients
• Practical Relative Order Attack in Deep Ranking
• GoTube: Scalable Stochastic Verification of Continuous-Depth Models
• Mitigating Adversarial Attacks by Distributing Different Copies to Different Users
• Intrinsic Geometric Vulnerability of High-Dimensional Artificial Intelligence
• Adversarial Robustness Study of Convolutional Neural Network for Lumbar Disk Shape Reconstruction from MR images
• On Procedural Adversarial Noise Attack And Defense
• Robust Spammer Detection by Nash Reinforcement Learning
• MUTEN: Boosting Gradient-Based Adversarial Attacks via Mutant-Based Ensembles
• Recurrent Attention Model with Log-Polar Mapping is Robust against Adversarial Attacks
• Towards a Theoretical Understanding of the Robustness of Variational Autoencoders
• What Do Adversarially Robust Models Look At?
• Certifiably Robust Variational Autoencoders
• Understanding the Error in Evaluating Adversarial Robustness
• Towards Deep Learning Models Resistant to Large Perturbations
• Meta-Learning the Search Distribution of Black-Box Random Search Based Adversarial Attacks
• An Empirical Study on the Relation between Network Interpretability and Adversarial Robustness
• Lagrangian Objective Function Leads to Improved Unforeseen Attack Generalization in Adversarial Training
• A Distributionally Robust Optimization Method for Adversarial Multiple Kernel Learning
• Dynamic Defense Approach for Adversarial Robustness in Deep Neural Networks via Stochastic Ensemble Smoothed Model
• Query complexity of adversarial attacks
• Adaptive Image Transformations for Transfer-based Adversarial Attack
• Enhance CNN Robustness Against Noises for Classification of 12-Lead ECG with Variable Length
• The Human Visual System and Adversarial AI
• When Side-Channel Attacks Break the Black-Box Property of Embedded Artificial Intelligence
• Augmenting Model Robustness with Transformation-Invariant Attacks
• Representation Quality Of Neural Networks Links To Adversarial Attacks and Defences
• On the Structural Sensitivity of Deep Convolutional Networks to the Directions of Fourier Basis Functions
• Dynamic Efficient Adversarial Training Guided by Gradient Magnitude
• What Clinical Trials Can Teach Us about the Development of More Resilient AI for Cybersecurity
• Automated Decision-based Adversarial Attacks
• Tightening the Approximation Error of Adversarial Risk with Auto Loss Function Search
• Contextual Fusion For Adversarial Robustness
• Robust Deep Neural Networks Inspired by Fuzzy Logic
• A note on hyperparameters in black-box adversarial examples
• Adversarially Robust and Explainable Model Compression with On-Device Personalization for Text Classification
• Luring of transferable adversarial perturbations in the black-box paradigm
• Regulatory Markets for AI Safety
• Semantics-Preserving Adversarial Training
• Harnessing Perceptual Adversarial Patches for Crowd Counting
• Multi-way Encoding for Robustness