On the Generalization Properties of Adversarial Training

Abstract

Modern machine learning and deep learning models are shown to be vulnerable when testing data are slightly perturbed. Existing theoretical studies of adversarial training algorithms mostly focus on either adversarial training losses or local convergence properties. In contrast, this paper studies the generalization performance of a generic adversarial training algorithm. Specifically, we consider linear regression models and two-layer neural networks (with lazy training) using squared loss under low-dimensional and high-dimensional regimes. In the former regime, after overcoming the non-smoothness of adversarial training, the adversarial risk of the trained models can converge to the minimal adversarial risk. In the latter regime, we discover that data interpolation prevents the adversarially robust estimator from being consistent. Therefore, inspired by successes of the least absolute shrinkage and selection operator (LASSO), we incorporate the L1 penalty in the high dimensional adversarial learning and show that it leads to consistent adversarially robust estimation. A series of numerical studies are conducted to demonstrate how the smoothness and L1 penalization help improve the adversarial robustness of DNN models.

References in corpus (18)

• Simultaneous analysis of Lasso and Dantzig selector
• Least squares after model selection in high-dimensional sparse models
• Adversarially Robust Generalization Requires More Data
• Fast is better than free: Revisiting adversarial training
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Understanding Adversarial Training: Increasing Local Stability of Neural Nets through Robust Optimization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers
• On the Convergence and Robustness of Adversarial Training
• Rademacher Complexity for Adversarially Robust Generalization
• The Robust Manifold Defense: Adversarial Training using Generative Models
• Smooth Adversarial Training
• Adversarially Robust Generalization Just Requires More Unlabeled Data
• Convergence of Adversarial Training in Overparametrized Neural Networks
• Overfitting in adversarially robust deep learning
• More Data Can Expand the Generalization Gap Between Adversarially Robust and Standard Models
• Towards Practical Lottery Ticket Hypothesis for Adversarial Training
• Rethinking Empirical Evaluation of Adversarial Robustness Using First-Order Attack Methods

Cited by in corpus (4)

• Recent Advances in Adversarial Training for Adversarial Robustness
• PAD: Towards Principled Adversarial Malware Detection Against Evasion Attacks
• Adversarially Robust Estimate and Risk Analysis in Linear Regression
• Asymptotic Behavior of Adversarial Training in Binary Classification