Bag of Tricks for Adversarial Training

Abstract

Adversarial training (AT) is one of the most effective strategies for promoting model robustness. However, recent benchmarks show that most of the proposed improvements on AT are less effective than simply early stopping the training procedure. This counter-intuitive fact motivates us to investigate the implementation details of tens of AT methods. Surprisingly, we find that the basic settings (e.g., weight decay, training schedule, etc.) used in these methods are highly inconsistent. In this work, we provide comprehensive evaluations on CIFAR-10, focusing on the effects of mostly overlooked training tricks and hyperparameters for adversarially trained models. Our empirical observations suggest that adversarial robustness is much more sensitive to some basic training settings than we thought. For example, a slightly different value of weight decay can reduce the model robust accuracy by more than 7%, which is probable to override the potential promotion induced by the proposed methods. We conclude a baseline training setting and re-implement previous defenses to achieve new state-of-the-art results. These facts also appeal to more concerns on the overlooked confounders when benchmarking defenses.

arXiv commentICLR 2021

References in corpus (40)

• Decoupled Weight Decay Regularization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models
• Theoretically Principled Trade-off between Robustness and Accuracy
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Certified Adversarial Robustness via Randomized Smoothing
• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
• On Evaluating Adversarial Robustness
• Fast is better than free: Revisiting adversarial training
• Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
• Detecting Adversarial Samples from Artifacts
• Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation
• On Detecting Adversarial Perturbations
• Adversarial Robustness through Local Linearization
• Improving Adversarial Robustness via Promoting Ensemble Diversity
• On the Convergence and Robustness of Adversarial Training
• Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNets
• Improving Black-box Adversarial Attacks with a Transfer-based Prior
• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
• Smooth Adversarial Training
• MMA Training: Direct Input Space Margin Maximization through Adversarial Training
• Self-Adaptive Training: beyond Empirical Risk Minimization
• Training verified learners with learned verifiers
• Adversarially Robust Generalization Just Requires More Unlabeled Data
• Instance adaptive adversarial training: Improved accuracy tradeoffs in neural nets
• Controlling Neural Level Sets
• Metric Learning for Adversarial Robustness
• Max-Mahalanobis Linear Discriminant Analysis Networks
• Label Smoothing and Logit Squeezing: A Replacement for Adversarial Training?
• Adversarial Distributional Training for Robust Deep Learning
• Dual Path Networks
• Curriculum Adversarial Training
• Learning to Defend by Learning to Attack
• Towards Understanding Fast Adversarial Training
• A Direct Approach to Robust Deep Learning Using Adversarial Networks
• RayS: A Ray Searching Method for Hard-label Adversarial Attack
• Using Single-Step Adversarial Training to Defend Iterative Adversarial Examples
• Exploring Model Robustness with Adaptive Networks and Improved Adversarial Training

Cited by in corpus (8)

• Boosting Adversarial Training with Hypersphere Embedding
• Recent Advances in Adversarial Training for Adversarial Robustness
• Probabilistic Margins for Instance Reweighting in Adversarial Training
• Improving Robustness using Generated Data
• Long-term Cross Adversarial Training: A Robust Meta-learning Method for Few-shot Classification Tasks
• Domain Invariant Adversarial Learning
• Dual Head Adversarial Training
• Bridged Adversarial Training