Enhance Diffusion to Improve Robust Generalization
arXiv:2306.02618 · doi:10.1145/3580305.3599333
Abstract
Deep neural networks are susceptible to human imperceptible adversarial perturbations. One of the strongest defense mechanisms is \emph{Adversarial Training} (AT). In this paper, we aim to address two predominant problems in AT. First, there is still little consensus on how to set hyperparameters with a performance guarantee for AT research, and customized settings impede a fair comparison between different model designs in AT research. Second, the robustly trained neural networks struggle to generalize well and suffer from tremendous overfitting. This paper focuses on the primary AT framework - Projected Gradient Descent Adversarial Training (PGD-AT). We approximate the dynamic of PGD-AT by a continuous-time Stochastic Differential Equation (SDE), and show that the diffusion term of this SDE determines the robust generalization. An immediate implication of this theoretical finding is that robust generalization is positively correlated with the ratio between learning rate and batch size. We further propose a novel approach, \emph{Diffusion Enhanced Adversarial Training} (DEAT), to manipulate the diffusion term to improve robust generalization with virtually no extra computational burden. We theoretically show that DEAT obtains a tighter generalization bound than PGD-AT. Our empirical investigation is extensive and firmly attests that DEAT universally outperforms PGD-AT by a significant margin.
arXiv commentAccepted at KDD 2023
References in corpus (33)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Explaining and Harnessing Adversarial Examples
- Intriguing properties of neural networks
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Theoretically Principled Trade-off between Robustness and Accuracy
- Visualizing the Loss Landscape of Neural Nets
- Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
- Don't Decay the Learning Rate, Increase the Batch Size
- Adversarially Robust Generalization Requires More Data
- Fast is better than free: Revisiting adversarial training
- Adversarial Attacks and Defences: A Survey
- Using Pre-Training Can Improve Model Robustness and Uncertainty
- Three Factors Influencing Minima in SGD
- On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems
- On the Convergence and Robustness of Adversarial Training
- Are Labels Required for Improving Adversarial Robustness?
- Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Training
- Stochastic Gradient Descent as Approximate Bayesian Inference
- Rethinking Softmax Cross-Entropy Loss for Adversarial Robustness
- MMA Training: Direct Input Space Margin Maximization through Adversarial Training
- Escaping Saddles with Stochastic Gradients
- Controlling Neural Level Sets
- Comprehensive Fair Meta-learned Recommender System
- Width Provably Matters in Optimization for Deep Linear Neural Networks
- Revisiting and Advancing Fast Adversarial Training Through The Lens of Bi-Level Optimization
- Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization
- RetrievalGuard: Provably Robust 1-Nearest Neighbor Image Retrieval
- Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent
- Adversarial Weight Perturbation Improves Generalization in Graph Neural Networks
- MLP Fusion: Towards Efficient Fine-tuning of Dense and Mixture-of-Experts Language Models
- PlayBest: Professional Basketball Player Behavior Synthesis via Planning with Diffusion
- Exploring Model Robustness with Adaptive Networks and Improved Adversarial Training