On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them

Abstract

We analyze the influence of adversarial training on the loss landscape of machine learning models. To this end, we first provide analytical studies of the properties of adversarial loss functions under different adversarial budgets. We then demonstrate that the adversarial loss landscape is less favorable to optimization, due to increased curvature and more scattered gradients. Our conclusions are validated by numerical analyses, which show that training under large adversarial budgets impede the escape from suboptimal random initialization, cause non-vanishing gradients and make the model find sharper minima. Based on these observations, we show that a periodic adversarial scheduling (PAS) strategy can effectively overcome these challenges, yielding better results than vanilla adversarial training while being much less sensitive to the choice of learning rate.

References in corpus (29)

• Adam: A Method for Stochastic Optimization
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Certified Adversarial Robustness via Randomized Smoothing
• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Certified Defenses against Adversarial Examples
• Understanding Adversarial Attacks on Deep Learning Based Medical Image Analysis Systems
• Adversarially Robust Generalization Requires More Data
• Fast is better than free: Revisiting adversarial training
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models
• Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers
• Essentially No Barriers in Neural Network Energy Landscape
• Are Labels Required for Improving Adversarial Robustness?
• Sharp Minima Can Generalize For Deep Nets
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Gradient Descent Converges to Minimizers
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• Hessian-based Analysis of Large Batch Training and Robustness to Adversaries
• A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation
• Provable Robustness of ReLU networks via Maximization of Linear Regions
• Model Compression with Adversarial Robustness: A Unified Optimization Framework
• Overfitting in adversarially robust deep learning
• Towards Stable and Efficient Training of Verifiably Robust Neural Networks
• Harnessing the Vulnerability of Latent Layers in Adversarially Trained Models
• Loss Patterns of Neural Networks

Cited by in corpus (9)

• SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks
• SAT: Improving Adversarial Training via Curriculum-Based Loss Smoothing
• Exploring Memorization in Adversarial Training
• Sharpness-aware Quantization for Deep Neural Networks
• On the Noise Stability and Robustness of Adversarially Trained Networks on NVM Crossbars
• Smoothness Analysis of Adversarial Training
• Training Provably Robust Models by Polyhedral Envelope Regularization
• Bridged Adversarial Training