Smoothness Analysis of Adversarial Training
arXiv:2103.01400
Abstract
Deep neural networks are vulnerable to adversarial attacks. Recent studies about adversarial robustness focus on the loss landscape in the parameter space since it is related to optimization and generalization performance. These studies conclude that the difficulty of adversarial training is caused by the non-smoothness of the loss function: i.e., its gradient is not Lipschitz continuous. However, this analysis ignores the dependence of adversarial attacks on model parameters. Since adversarial attacks are optimized for models, they should depend on the parameters. Considering this dependence, we analyze the smoothness of the loss function of adversarial training using the optimal attacks for the model parameter in more detail. We reveal that the constraint of adversarial attacks is one cause of the non-smoothness and that the smoothness depends on the types of the constraints. Specifically, the constraint can cause non-smoothness more than the constraint. Moreover, our analysis implies that if we flatten the loss function with respect to input data, the Lipschitz constant of the gradient of adversarial loss tends to increase. To address the non-smoothness, we show that EntropySGD smoothens the non-smooth loss and improves the performance of adversarial training.
arXiv commentThe latest version of this article is published in IEEE Transactions on Neural Networks and Learning Systems (DOI: 10.1109/TNNLS.2023.3244172). 22 pages, 7 figures
References in corpus (16)
- Wide Residual Networks
- Theoretically Principled Trade-off between Robustness and Accuracy
- Searching for Activation Functions
- Certified Adversarial Robustness via Randomized Smoothing
- Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
- Improving Generalization Performance by Switching from Adam to SGD
- Three Factors Influencing Minima in SGD
- Adversarial Robustness through Local Linearization
- Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks
- Sharp Minima Can Generalize For Deep Nets
- Evaluating and Understanding the Robustness of Adversarial Logit Pairing
- On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them
- Understanding Catastrophic Overfitting in Single-step Adversarial Training
- S-SGD: Symmetrical Stochastic Gradient Descent with Weight Noise Injection for Reaching Flat Minima
- Adversarial Training Makes Weight Loss Landscape Sharper in Logistic Regression
- Why flatness does and does not correlate with generalization for deep neural networks