Unlabeled Data Improves Adversarial Robustness

Abstract

We demonstrate, theoretically and empirically, that adversarial robustness can significantly benefit from semisupervised learning. Theoretically, we revisit the simple Gaussian model of Schmidt et al. that shows a sample complexity gap between standard and robust classification. We prove that unlabeled data bridges this gap: a simple semisupervised learning procedure (self-training) achieves high robust accuracy using the same number of labels required for achieving high standard accuracy. Empirically, we augment CIFAR-10 with 500K unlabeled images sourced from 80 Million Tiny Images and use robust self-training to outperform state-of-the-art robust accuracies by over 5 points in (i) ${\ell }_{\infty }$ robustness against several strong attacks via adversarial training and (ii) certified ${\ell }_2$ and ${\ell }_{\infty }$ robustness via randomized smoothing. On SVHN, adding the dataset's own extra training set with the labels removed provides gains of 4 to 10 points, within 1 point of the gain from using the extra labels.

arXiv commentCorrected some math typos in the proof of Lemma 1

References in corpus (24)

• Improved Regularization of Convolutional Neural Networks with Cutout
• Unsupervised Data Augmentation for Consistency Training
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Temporal Ensembling for Semi-Supervised Learning
• Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Theoretically Principled Trade-off between Robustness and Accuracy
• AutoAugment: Learning Augmentation Policies from Data
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Certified Adversarial Robustness via Randomized Smoothing
• Certified Defenses against Adversarial Examples
• Adversarially Robust Generalization Requires More Data
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
• Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
• On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models
• Semidefinite relaxations for certifying robustness to adversarial examples
• Are Labels Required for Improving Adversarial Robustness?
• Adversarial Training Can Hurt Generalization
• Adversarial examples from computational constraints
• Rademacher Complexity for Adversarially Robust Generalization
• Adversarially Robust Generalization Just Requires More Unlabeled Data
• Evaluating and Understanding the Robustness of Adversarial Logit Pairing
• VC Classes are Adversarially Robustly Learnable, but Only Improperly

Cited by in corpus (21)

• Unsupervised Data Augmentation for Consistency Training
• Theoretically Principled Trade-off between Robustness and Accuracy
• Self-training with Noisy Student improves ImageNet classification
• Are Labels Required for Improving Adversarial Robustness?
• Adversarial Training Can Hurt Generalization
• Adversarially Robust Generalization Just Requires More Unlabeled Data
• Snippext: Semi-supervised Opinion Mining with Augmented Data
• Overfitting in adversarially robust deep learning
• A survey of algorithmic recourse: definitions, formulations, solutions, and prospects
• Natural Language Adversarial Defense through Synonym Encoding
• Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
• Interpolated Adversarial Training: Achieving Robust Neural Networks without Sacrificing Too Much Accuracy
• Deep Neural Network Fingerprinting by Conferrable Adversarial Examples
• When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks
• Efficient Adversarial Training with Transferable Adversarial Examples
• Imbalanced Gradients: A Subtle Cause of Overestimated Adversarial Robustness
• Calibration and Consistency of Adversarial Surrogate Losses
• CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothing
• Asymptotic Behavior of Adversarial Training in Binary Classification
• Robust Sensible Adversarial Learning of Deep Neural Networks for Image Classification
• Domain Invariant Adversarial Learning