Adversarially Robust Generalization Just Requires More Unlabeled Data

Abstract

Neural network robustness has recently been highlighted by the existence of adversarial examples. Many previous works show that the learned networks do not perform well on perturbed test data, and significantly more labeled data is required to achieve adversarially robust generalization. In this paper, we theoretically and empirically show that with just more unlabeled data, we can learn a model with better adversarially robust generalization. The key insight of our results is based on a risk decomposition theorem, in which the expected robust risk is separated into two parts: the stability part which measures the prediction stability in the presence of perturbations, and the accuracy part which evaluates the standard classification accuracy. As the stability part does not depend on any label information, we can optimize this part using unlabeled data. We further prove that for a specific Gaussian mixture problem, adversarially robust generalization can be almost as easy as the standard generalization in supervised learning if a sufficiently large amount of unlabeled data is provided. Inspired by the theoretical findings, we further show that a practical adversarial training algorithm that leverages unlabeled data can improve adversarial robust generalization on MNIST and Cifar-10.

16 pages. Submitted to ICLR 2020

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Theoretically Principled Trade-off between Robustness and Accuracy
• Semi-Supervised Learning with Ladder Networks
• Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning
• Adversarially Robust Generalization Requires More Data
• Robustness May Be at Odds with Accuracy
• Distributional Smoothing with Virtual Adversarial Training
• Learning with a Strong Adversary
• Adversarial vulnerability for any classifier
• There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average
• You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle
• Unlabeled Data Improves Adversarial Robustness
• Convergence of Adversarial Training in Overparametrized Neural Networks
• PAC-learning in the presence of evasion adversaries

Cited by in corpus (53)

• Unsupervised Data Augmentation for Consistency Training
• Theoretically Principled Trade-off between Robustness and Accuracy
• Self-training with Noisy Student improves ImageNet classification
• Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks
• Adversarial Weight Perturbation Helps Robust Generalization
• Are Labels Required for Improving Adversarial Robustness?
• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
• Adversarial Training Can Hurt Generalization
• Fixing Data Augmentation to Improve Adversarial Robustness
• A Closer Look at Accuracy vs. Robustness
• Robust Pre-Training by Adversarial Contrastive Learning
• Unlabeled Data Improves Adversarial Robustness
• Boosting Adversarial Training with Hypersphere Embedding
• MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius
• Robust Local Features for Improving the Generalization of Adversarial Training
• Adversarial Examples Improve Image Recognition
• Overfitting in adversarially robust deep learning
• MaxMatch: Semi-Supervised Learning with Worst-Case Consistency
• Adversarial Distributional Training for Robust Deep Learning
• Natural Language Adversarial Defense through Synonym Encoding
• Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
• Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks
• The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalization
• Interpolated Adversarial Training: Achieving Robust Neural Networks without Sacrificing Too Much Accuracy
• Black-Box Certification with Randomized Smoothing: A Functional Optimization Based Framework
• Intriguing properties of adversarial training at scale
• Statistical and Algorithmic Insights for Semi-supervised Learning with Self-training
• Adaptive Self-Training for Object Detection
• Exploring Memorization in Adversarial Training
• Bag of Tricks for Adversarial Training
• ATOM: Robustifying Out-of-distribution Detection Using Outlier Mining
• Understanding Generalization in Adversarial Training via the Bias-Variance Decomposition
• On the Generalization Properties of Adversarial Training
• PDA: Progressive Data Augmentation for General Robustness of Deep Neural Networks
• Stochastic Variance Reduced Ensemble Adversarial Attack for Boosting the Adversarial Transferability
• Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?
• Improving Global Adversarial Robustness Generalization With Adversarially Trained GAN
• Train simultaneously, generalize better: Stability of gradient-based minimax learners
• Sharp Statistical Guarantees for Adversarially Robust Gaussian Classification
• Beyond Perturbations: Learning Guarantees with Arbitrary Adversarial Test Examples
• Improving Adversarial Robustness via Unlabeled Out-of-Domain Data
• Improving Robustness using Generated Data
• Adversarially Robust Estimate and Risk Analysis in Linear Regression
• Adversarial Robustness of Stabilized NeuralODEs Might be from Obfuscated Gradients
• Improving Model Robustness with Latent Distribution Locally and Globally
• Learning Diverse-Structured Networks for Adversarial Robustness
• Towards Deep Learning Models Resistant to Large Perturbations
• AlphaMatch: Improving Consistency for Semi-supervised Learning with Alpha-divergence
• Multiplicative Reweighting for Robust Neural Network Optimization
• Domain Invariant Adversarial Learning
• Model-Agnostic Meta-Attack: Towards Reliable Evaluation of Adversarial Robustness
• Adversarial Training with Stochastic Weight Average
• Bridged Adversarial Training