The Space of Transferable Adversarial Examples

Abstract

Adversarial examples are maliciously perturbed inputs designed to mislead machine learning (ML) models at test-time. They often transfer: the same adversarial example fools more than one model. In this work, we propose novel methods for estimating the previously unknown dimensionality of the space of adversarial inputs. We find that adversarial examples span a contiguous subspace of large (~25) dimensionality. Adversarial subspaces with higher dimensionality are more likely to intersect. We find that for two different models, a significant fraction of their subspaces is shared, thus enabling transferability. In the first quantitative analysis of the similarity of different models' decision boundaries, we show that these boundaries are actually close in arbitrary directions, whether adversarial or benign. We conclude by formally studying the limits of transferability. We derive (1) sufficient conditions on the data distribution that imply transferability for simple model classes and (2) examples of scenarios in which transfer does not occur. These findings indicate that it may be possible to design defenses against transfer-based attacks, even for models that are vulnerable to direct attacks.

15 pages, 7 figures

References in corpus (6)

• Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
• Stealing Machine Learning Models via Prediction APIs
• Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
• Generating Adversarial Malware Examples for Black-Box Attacks Based on GAN
• Adversarial Perturbations Against Deep Neural Networks for Malware Classification
• Practical Black-Box Attacks against Machine Learning

Cited by in corpus (156)

• Ensemble Adversarial Training: Attacks and Defenses
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Adversarial Attacks on Neural Networks for Graph Data
• Understanding Adversarial Attacks on Deep Learning Based Medical Image Analysis Systems
• Adversarially Robust Generalization Requires More Data
• Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality
• Adversarial Examples Are Not Bugs, They Are Features
• PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples
• Adversarial Examples: Opportunities and Challenges
• Generating Adversarial Examples with Adversarial Networks
• Adversarial Examples: Attacks and Defenses for Deep Learning
• Stochastic Activation Pruning for Robust Adversarial Defense
• Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey
• Adversarial Machine Learning in Image Classification: A Survey Towards the Defender's Perspective
• Transferable Clean-Label Poisoning Attacks on Deep Neural Nets
• How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defenses
• Adversarial Attack Vulnerability of Medical Image Analysis Systems: Unexplored Factors
• Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks
• Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Training
• The Robust Manifold Defense: Adversarial Training using Generative Models
• Guessing Smart: Biased Sampling for Efficient Black-Box Adversarial Attacks
• The Butterfly Effect in Artificial Intelligence Systems: Implications for AI Bias and Fairness
• Adversarial Attacks Against Medical Deep Learning Systems
• Blind Backdoors in Deep Learning Models
• Why Do Adversarial Attacks Transfer? Explaining Transferability of Evasion and Poisoning Attacks
• Understanding and Enhancing the Transferability of Adversarial Examples
• The Pitfalls of Simplicity Bias in Neural Networks
• Improving the Generalization of Adversarial Training with Domain Adaptation
• Extending Defensive Distillation
• Exploring the Space of Black-box Attacks on Deep Neural Networks
• Partial success in closing the gap between human and machine vision
• DVERGE: Diversifying Vulnerabilities for Enhanced Robust Generation of Ensembles
• Security Analysis of Deep Neural Networks Operating in the Presence of Cache Side-Channel Attacks
• A Survey of Game Theoretic Approaches for Adversarial Machine Learning in Cybersecurity Tasks
• Adversarial Examples in Modern Machine Learning: A Review
• Walking on the Edge: Fast, Low-Distortion Adversarial Examples
• Adversarial Examples - A Complete Characterisation of the Phenomenon
• NATTACK: Learning the Distributions of Adversarial Examples for an Improved Black-Box Attack on Deep Neural Networks
• Recent Advances in Adversarial Training for Adversarial Robustness
• Denoised Smoothing: A Provable Defense for Pretrained Classifiers
• Do Wider Neural Networks Really Help Adversarial Robustness?
• Perturbing Across the Feature Hierarchy to Improve Standard and Strict Blackbox Attack Transferability
• Adversarial Attack Framework on Graph Embedding Models with Limited Knowledge
• GAMIN: An Adversarial Approach to Black-Box Model Inversion
• The Threat of Adversarial Attacks on Machine Learning in Network Security -- A Survey
• Adversarial Examples in Deep Learning: Characterization and Divergence
• Task dependent Deep LDA pruning of neural networks
• Mutual Adversarial Training: Learning together is better than going alone
• Exploring Connections Between Active Learning and Model Extraction
• Deep Neural Network Fingerprinting by Conferrable Adversarial Examples
• Characterizing Bias in Classifiers using Generative Models
• The Faults in Our Pi Stars: Security Issues and Open Challenges in Deep Reinforcement Learning
• Fooling a Real Car with Adversarial Traffic Signs
• Beneficial Perturbation Network for designing general adaptive artificial intelligence systems
• A Panda? No, It's a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network Inference
• To compress or not to compress: Understanding the Interactions between Adversarial Attacks and Neural Network Compression
• Adversarially Robust Training through Structured Gradient Regularization
• Membership Leakage in Label-Only Exposures
• Breaking Transferability of Adversarial Samples with Randomness
• Attacking Adversarial Attacks as A Defense
• Curls & Whey: Boosting Black-Box Adversarial Attacks
• Learning deep forest with multi-scale Local Binary Pattern features for face anti-spoofing
• Security and Privacy for Artificial Intelligence: Opportunities and Challenges
• Transferable Universal Adversarial Perturbations Using Generative Models
• Robustifying Models Against Adversarial Attacks by Langevin Dynamics
• Overcoming catastrophic forgetting problem by weight consolidation and long-term memory
• GreedyFool: Distortion-Aware Sparse Adversarial Attack
• Adversarial Examples on Object Recognition: A Comprehensive Survey
• Adversarial Robustness via Label-Smoothing
• Towards Security Threats of Deep Learning Systems: A Survey
• Subverting Fair Image Search with Generative Adversarial Perturbations
• Snooping Attacks on Deep Reinforcement Learning
• Toward Improving Synthetic Audio Spoofing Detection Robustness via Meta-Learning and Disentangled Training With Adversarial Examples
• Adversarial Binaries for Authorship Identification
• Fooling Vision and Language Models Despite Localization and Attention Mechanism
• Nonlinear Projection Based Gradient Estimation for Query Efficient Blackbox Attacks
• Understanding the Robustness of Skeleton-based Action Recognition under Adversarial Attack
• SoK: Systematic analysis of adversarial threats against deep learning approaches for autonomous anomaly detection systems in SDN-IoT networks
• GeoDA: a geometric framework for black-box adversarial attacks
• TRS: Transferability Reduced Ensemble via Encouraging Gradient Diversity and Model Smoothness
• On the Effect of Low-Rank Weights on Adversarial Robustness of Neural Networks
• BASAR:Black-box Attack on Skeletal Action Recognition
• Detecting Adversarial Examples from Sensitivity Inconsistency of Spatial-Transform Domain
• On the Effectiveness of Regularization Against Membership Inference Attacks
• SOAR: Second-Order Adversarial Regularization
• A Multiversion Programming Inspired Approach to Detecting Audio Adversarial Examples
• Robustness Certificates Against Adversarial Examples for ReLU Networks
• The Adversarial Attack and Detection under the Fisher Information Metric
• Neural Networks with Structural Resistance to Adversarial Attacks
• SoK: How Robust is Image Classification Deep Neural Network Watermarking? (Extended Version)
• Progressive-Scale Boundary Blackbox Attack via Projective Gradient Estimation
• SMART: Skeletal Motion Action Recognition aTtack
• Detecting Adversarial Examples - A Lesson from Multimedia Forensics
• Stochastic Combinatorial Ensembles for Defending Against Adversarial Examples
• Adversarial Examples in Constrained Domains
• Open DNN Box by Power Side-Channel Attack
• Predicting Adversarial Examples with High Confidence
• One Bit Matters: Understanding Adversarial Examples as the Abuse of Redundancy
• AutoGAN: Robust Classifier Against Adversarial Attacks
• Intermediate Level Adversarial Attack for Enhanced Transferability
• Crafting Adversarial Examples for Deep Learning Based Prognostics (Extended Version)
• Toward Metrics for Differentiating Out-of-Distribution Sets
• Active Deep Learning Attacks under Strict Rate Limitations for Online API Calls
• Minimax Defense against Gradient-based Adversarial Attacks
• Man-in-the-Middle Attacks against Machine Learning Classifiers via Malicious Generative Models
• Analytical Moment Regularizer for Gaussian Robust Networks
• Reducing Adversarial Example Transferability Using Gradient Regularization
• Beneficial Perturbations Network for Defending Adversarial Examples
• Ensemble Noise Simulation to Handle Uncertainty about Gradient-based Adversarial Attacks
• Mixed Nash Equilibria in the Adversarial Examples Game
• On the Exploitability of Audio Machine Learning Pipelines to Surreptitious Adversarial Examples
• Towards Adversarial Robustness via Transductive Learning
• ANCER: Anisotropic Certification via Sample-wise Volume Maximization
• Uncovering the Connections Between Adversarial Transferability and Knowledge Transferability
• Implicit Generative Modeling of Random Noise during Training for Adversarial Robustness
• Adversarial Robustness on Image Classification with $k$-means
• Traversing the Subspace of Adversarial Patches
• Simulating Unknown Target Models for Query-Efficient Black-box Attacks
• Dynamic Efficient Adversarial Training Guided by Gradient Magnitude
• Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)
• Robust and Information-theoretically Safe Bias Classifier against Adversarial Attacks
• On the Structural Sensitivity of Deep Convolutional Networks to the Directions of Fourier Basis Functions
• Adv-4-Adv: Thwarting Changing Adversarial Perturbations via Adversarial Domain Adaptation
• Unifying Bilateral Filtering and Adversarial Training for Robust Neural Networks
• Analysis of Random Perturbations for Robust Convolutional Neural Networks
• Beneficial perturbation network for continual learning
• Feedback Learning for Improving the Robustness of Neural Networks
• Deep Bayesian Image Set Classification: A Defence Approach against Adversarial Attacks
• CoG: a Two-View Co-training Framework for Defending Adversarial Attacks on Graph
• Multi-concept adversarial attacks
• Defense-guided Transferable Adversarial Attacks
• Adversarial Interaction Attack: Fooling AI to Misinterpret Human Intentions
• Target Training Does Adversarial Training Without Adversarial Samples
• Understanding Robustness in Teacher-Student Setting: A New Perspective
• Sparse Coding Frontend for Robust Neural Networks
• Enhancing Resilience of Deep Learning Networks by Means of Transferable Adversaries
• Tricking Adversarial Attacks To Fail
• Adversarial Attacks for Multi-view Deep Models
• Certified Robustness of Graph Classification against Topology Attack with Randomized Smoothing
• HAWKEYE: Adversarial Example Detector for Deep Neural Networks
• On the Matrix-Free Generation of Adversarial Perturbations for Black-Box Attacks
• A New Family of Neural Networks Provably Resistant to Adversarial Attacks
• Generating Adversarial Examples With Conditional Generative Adversarial Net
• On the Vulnerability of CNN Classifiers in EEG-Based BCIs
• Learning To Characterize Adversarial Subspaces
• Beyond Universal Person Re-ID Attack
• MGA: Momentum Gradient Attack on Network
• PoisHygiene: Detecting and Mitigating Poisoning Attacks in Neural Networks
• Investigating Image Applications Based on Spatial-Frequency Transform and Deep Learning Techniques
• Defense Through Diverse Directions
• Do Deep Minds Think Alike? Selective Adversarial Attacks for Fine-Grained Manipulation of Multiple Deep Neural Networks
• Training Machine Learning Models by Regularizing their Explanations
• Learning Task-aware Robust Deep Learning Systems
• A general metric for identifying adversarial images
• Delving into the pixels of adversarial samples
• Understanding Adversarial Examples Through Deep Neural Network's Response Surface and Uncertainty Regions