Are Labels Required for Improving Adversarial Robustness?

Abstract

Recent work has uncovered the interesting (and somewhat surprising) finding that training models to be invariant to adversarial perturbations requires substantially larger datasets than those required for standard classification. This result is a key hurdle in the deployment of robust machine learning models in many real world applications where labeled data is expensive. Our main insight is that unlabeled data can be a competitive alternative to labeled data for training adversarially robust models. Theoretically, we show that in a simple statistical setting, the sample complexity for learning an adversarially robust model from unlabeled data matches the fully supervised case up to constant factors. On standard datasets like CIFAR-10, a simple Unsupervised Adversarial Training (UAT) approach using unlabeled data improves robust accuracy by 21.7% over using 4K supervised examples alone, and captures over 95% of the improvement from the same number of labeled examples. Finally, we report an improvement of 4% over the previous state-of-the-art on CIFAR-10 against the strongest known attack by using additional unlabeled data from the uncurated 80 Million Tiny Images dataset. This demonstrates that our finding extends as well to the more realistic case where unlabeled data is also uncurated, therefore opening a new avenue for improving adversarial training.

Appears in the Thirty-Third Annual Conference on Neural Information Processing Systems (NeurIPS 2019)

References in corpus (25)

• Adam: A Method for Stochastic Optimization
• Explaining and Harnessing Adversarial Examples
• End to End Learning for Self-Driving Cars
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Ensemble Adversarial Training: Attacks and Defenses
• Wide Residual Networks
• Unsupervised Data Augmentation for Consistency Training
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Theoretically Principled Trade-off between Robustness and Accuracy
• Delving into Transferable Adversarial Examples and Black-box Attacks
• Towards Deep Neural Network Architectures Robust to Adversarial Examples
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• Adversarially Robust Generalization Requires More Data
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Deep Learning is Robust to Massive Label Noise
• Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
• Learning with Pseudo-Ensembles
• PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples
• On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models
• Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models
• Rademacher Complexity for Adversarially Robust Generalization
• Adversarially Robust Generalization Just Requires More Unlabeled Data
• Unlabeled Data Improves Adversarial Robustness
• An Alternative Surrogate Loss for PGD-based Adversarial Testing
• Adversarial Risk Bounds via Function Transformation

Cited by in corpus (54)

• Unsupervised Data Augmentation for Consistency Training
• Theoretically Principled Trade-off between Robustness and Accuracy
• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Self-training with Noisy Student improves ImageNet classification
• Adversarial Weight Perturbation Helps Robust Generalization
• Adversarial Training Can Hurt Generalization
• Rethinking Softmax Cross-Entropy Loss for Adversarial Robustness
• MMA Training: Direct Input Space Margin Maximization through Adversarial Training
• Adversarial Training against Location-Optimized Adversarial Patches
• Attacks Which Do Not Kill Training Make Adversarial Learning Stronger
• Adversarial Self-Supervised Contrastive Learning
• A Closer Look at Accuracy vs. Robustness
• Unlabeled Data Improves Adversarial Robustness
• Boosting Adversarial Training with Hypersphere Embedding
• MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius
• Towards Evaluating the Robustness of Deep Diagnostic Models by Adversarial Attack
• Understanding and Mitigating the Tradeoff Between Robustness and Accuracy
• Adversarial Examples Improve Image Recognition
• Diversity can be Transferred: Output Diversification for White- and Black-box Attacks
• On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them
• Confidence-Calibrated Adversarial Training: Generalizing to Unseen Attacks
• Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
• Adversarial Distributional Training for Robust Deep Learning
• Natural Language Adversarial Defense through Synonym Encoding
• Adversarial Robustness via Fisher-Rao Regularization
• When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks
• InfoAT: Improving Adversarial Training Using the Information Bottleneck Principle
• Intriguing properties of adversarial training at scale
• Understanding Generalization in Adversarial Training via the Bias-Variance Decomposition
• Manifold Regularization for Locally Stable Deep Neural Networks
• ATOM: Robustifying Out-of-distribution Detection Using Outlier Mining
• Stochastic Security: Adversarial Defense Using Long-Run Dynamics of Energy-Based Models
• Bit Error Robustness for Energy-Efficient DNN Accelerators
• Imbalanced Gradients: A Subtle Cause of Overestimated Adversarial Robustness
• Sharp Statistical Guarantees for Adversarially Robust Gaussian Classification
• Improving Global Adversarial Robustness Generalization With Adversarially Trained GAN
• Defending against adversarial attacks on medical imaging AI system, classification or detection?
• Self-training Avoids Using Spurious Features Under Domain Shift
• Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better
• Beyond Perturbations: Learning Guarantees with Arbitrary Adversarial Test Examples
• Improving Adversarial Robustness via Unlabeled Out-of-Domain Data
• Toward Adversarial Robustness via Semi-supervised Robust Training
• Beneficial Perturbations Network for Defending Adversarial Examples
• A Useful Taxonomy for Adversarial Robustness of Neural Networks
• Domain Invariant Adversarial Learning
• Removing Undesirable Feature Contributions Using Out-of-Distribution Data
• Enhance Diffusion to Improve Robust Generalization
• Bridged Adversarial Training
• Decoder-free Robustness Disentanglement without (Additional) Supervision
• Derivation of Information-Theoretically Optimal Adversarial Attacks with Applications to Robust Machine Learning
• Adversarial Training with Stochastic Weight Average
• Adversarial Training: embedding adversarial perturbations into the parameter space of a neural network to build a robust system
• Training Provably Robust Models by Polyhedral Envelope Regularization