The Limitations of Adversarial Training and the Blind-Spot Attack

Abstract

The adversarial training procedure proposed by Madry et al. (2018) is one of the most effective methods to defend against adversarial examples in deep neural networks (DNNs). In our paper, we shed some lights on the practicality and the hardness of adversarial training by showing that the effectiveness (robustness on test set) of adversarial training has a strong correlation with the distance between a test point and the manifold of training data embedded by the network. Test examples that are relatively far away from this manifold are more likely to be vulnerable to adversarial attacks. Consequentially, an adversarial training based defense is susceptible to a new class of attacks, the "blind-spot attack", where the input images reside in "blind-spots" (low density regions) of the empirical distribution of training data but is still on the ground-truth data manifold. For MNIST, we found that these blind-spots can be easily found by simply scaling and shifting image pixel values. Most importantly, for large datasets with high dimensional and complex data manifold (CIFAR, ImageNet, etc), the existence of blind-spots in adversarial training makes defending on any valid test examples difficult due to the curse of dimensionality and the scarcity of training data. Additionally, we find that blind-spots also exist on provable defenses including (Wong & Kolter, 2018) and (Sinha et al., 2018) because these trainable robustness certificates can only be practically optimized on a limited set of training data.

arXiv commentAccepted by International Conference on Learning Representations (ICLR) 2019. Huan Zhang and Hongge Chen contributed equally

References in corpus (20)

• Explaining and Harnessing Adversarial Examples
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Synthesizing Robust Adversarial Examples
• Certified Defenses against Adversarial Examples
• Adversarially Robust Generalization Requires More Data
• Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality
• Countering Adversarial Images using Input Transformations
• PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples
• Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models
• Generating Adversarial Examples with Adversarial Networks
• Stochastic Activation Pruning for Robust Adversarial Defense
• Mitigating Adversarial Effects Through Randomization
• A Dual Approach to Scalable Verification of Deep Networks
• Adversarial examples from computational constraints
• Constructing Unrestricted Adversarial Examples with Generative Models
• PAC-learning in the presence of evasion adversaries
• Adversarial Examples from Cryptographic Pseudo-Random Generators
• Distributionally Adversarial Attack

Cited by in corpus (21)

• Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Training
• MMA Training: Direct Input Space Margin Maximization through Adversarial Training
• Functional Adversarial Attacks
• Adversarial Training against Location-Optimized Adversarial Patches
• Robust Local Features for Improving the Generalization of Adversarial Training
• Towards Stable and Efficient Training of Verifiably Robust Neural Networks
• Sign-OPT: A Query-Efficient Hard-label Adversarial Attack
• GenAttack: Practical Black-box Attacks with Gradient-Free Optimization
• Perceptual Adversarial Robustness: Defense Against Unseen Threat Models
• Brain Programming is Immune to Adversarial Attacks: Towards Accurate and Robust Image Classification using Symbolic Learning
• Adversarial Defense by Latent Style Transformations
• Efficient Adversarial Training with Transferable Adversarial Examples
• Adversarial Attacks and Defenses for Speech Recognition Systems
• Pelta: Shielding Transformers to Mitigate Evasion Attacks in Federated Learning
• Adversarial and Natural Perturbations for General Robustness
• Local Competition and Uncertainty for Adversarial Robustness in Deep Learning
• Local Competition and Stochasticity for Adversarial Robustness in Deep Learning
• Unsupervised Cross-Lingual Speech Emotion Recognition Using Pseudo Multilabel
• Defense Through Diverse Directions
• Understanding Robustness in Teacher-Student Setting: A New Perspective
• Wiggling Weights to Improve the Robustness of Classifiers