Understanding and Mitigating the Tradeoff Between Robustness and Accuracy

Abstract

Adversarial training augments the training set with perturbations to improve the robust error (over worst-case perturbations), but it often leads to an increase in the standard error (on unperturbed test inputs). Previous explanations for this tradeoff rely on the assumption that no predictor in the hypothesis class has low standard and robust error. In this work, we precisely characterize the effect of augmentation on the standard error in linear regression when the optimal linear predictor has zero standard and robust error. In particular, we show that the standard error could increase even when the augmented perturbations have noiseless observations from the optimal linear predictor. We then prove that the recently proposed robust self-training (RST) estimator improves robust error without sacrificing standard error for noiseless linear regression. Empirically, for neural networks, we find that RST with different adversarial training methods improves both standard and robust error for random and adversarial rotations and adversarial ${\ell }_{\infty }$ perturbations in CIFAR-10.

arXiv commentAppearing at International Conference on Machine Learning (ICML) 2020

References in corpus (12)

• Unsupervised Data Augmentation for Consistency Training
• Theoretically Principled Trade-off between Robustness and Accuracy
• Benign Overfitting in Linear Regression
• Adversarially Robust Generalization Requires More Data
• Exploring the Landscape of Spatial Robustness
• Are Labels Required for Improving Adversarial Robustness?
• The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
• A Fourier Perspective on Model Robustness in Computer Vision
• Adversarial Robustness May Be at Odds With Simplicity
• The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalization
• Interpolated Adversarial Training: Achieving Robust Neural Networks without Sacrificing Too Much Accuracy
• Intriguing Properties of Adversarial Examples

Cited by in corpus (27)

• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• RobustBench: a standardized adversarial robustness benchmark
• Adversarial Training for Large Neural Language Models
• Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data
• A Closer Look at Accuracy vs. Robustness
• Adversarial Examples Improve Image Recognition
• Do Wider Neural Networks Really Help Adversarial Robustness?
• More Data Can Expand the Generalization Gap Between Adversarially Robust and Standard Models
• Understanding Self-Training for Gradual Domain Adaptation
• WeMix: How to Better Utilize Data Augmentation
• On the Generalization Effects of Linear Transformations in Data Augmentation
• Understanding Generalization in Adversarial Training via the Bias-Variance Decomposition
• Revisiting Hilbert-Schmidt Information Bottleneck for Adversarial Robustness
• Decision-based Universal Adversarial Attack
• Sharp Statistical Guarantees for Adversarially Robust Gaussian Classification
• Pyramid Adversarial Training Improves ViT Performance
• Distinguishing rule- and exemplar-based generalization in learning systems
• Path classification by stochastic linear recurrent neural networks
• Bridging the Gap Between Adversarial Robustness and Optimization Bias
• Asymptotic Behavior of Adversarial Training in Binary Classification
• Representation Quality Of Neural Networks Links To Adversarial Attacks and Defences
• Learning Under Adversarial and Interventional Shifts
• Training Efficiency and Robustness in Deep Learning
• Unique properties of adversarially trained linear classifiers on Gaussian data
• Pareto Adversarial Robustness: Balancing Spatial Robustness and Sensitivity-based Robustness
• Classification and Adversarial examples in an Overparameterized Linear Model: A Signal Processing Perspective
• Noisy Feature Mixup