Unlabeled Data Improves Adversarial Robustness

Abstract

We demonstrate, theoretically and empirically, that adversarial robustness can significantly benefit from semisupervised learning. Theoretically, we revisit the simple Gaussian model of Schmidt et al. that shows a sample complexity gap between standard and robust classification. We prove that unlabeled data bridges this gap: a simple semisupervised learning procedure (self-training) achieves high robust accuracy using the same number of labels required for achieving high standard accuracy. Empirically, we augment CIFAR-10 with 500K unlabeled images sourced from 80 Million Tiny Images and use robust self-training to outperform state-of-the-art robust accuracies by over 5 points in (i) ${\ell }_{\infty }$ robustness against several strong attacks via adversarial training and (ii) certified ${\ell }_2$ and ${\ell }_{\infty }$ robustness via randomized smoothing. On SVHN, adding the dataset's own extra training set with the labels removed provides gains of 4 to 10 points, within 1 point of the gain from using the extra labels.

Corrected some math typos in the proof of Lemma 1

References in corpus (13)

• Improved Regularization of Convolutional Neural Networks with Cutout
• Unsupervised Data Augmentation for Consistency Training
• Temporal Ensembling for Semi-Supervised Learning
• Theoretically Principled Trade-off between Robustness and Accuracy
• Certified Adversarial Robustness via Randomized Smoothing
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models
• Semidefinite relaxations for certifying robustness to adversarial examples
• Are Labels Required for Improving Adversarial Robustness?
• Adversarial Training Can Hurt Generalization
• Rademacher Complexity for Adversarially Robust Generalization
• Adversarially Robust Generalization Just Requires More Unlabeled Data
• VC Classes are Adversarially Robustly Learnable, but Only Improperly

Cited by in corpus (21)

• Unsupervised Data Augmentation for Consistency Training
• Theoretically Principled Trade-off between Robustness and Accuracy
• Self-training with Noisy Student improves ImageNet classification
• Are Labels Required for Improving Adversarial Robustness?
• Adversarial Training Can Hurt Generalization
• Adversarially Robust Generalization Just Requires More Unlabeled Data
• Snippext: Semi-supervised Opinion Mining with Augmented Data
• Overfitting in adversarially robust deep learning
• A survey of algorithmic recourse: definitions, formulations, solutions, and prospects
• Natural Language Adversarial Defense through Synonym Encoding
• Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
• Interpolated Adversarial Training: Achieving Robust Neural Networks without Sacrificing Too Much Accuracy
• Deep Neural Network Fingerprinting by Conferrable Adversarial Examples
• When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks
• Efficient Adversarial Training with Transferable Adversarial Examples
• Imbalanced Gradients: A Subtle Cause of Overestimated Adversarial Robustness
• Calibration and Consistency of Adversarial Surrogate Losses
• CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothing
• Asymptotic Behavior of Adversarial Training in Binary Classification
• Robust Sensible Adversarial Learning of Deep Neural Networks for Image Classification
• Domain Invariant Adversarial Learning