Enhance Diffusion to Improve Robust Generalization
arXiv:2306.02618 · doi:10.1145/3580305.3599333
Abstract
Deep neural networks are susceptible to human imperceptible adversarial perturbations. One of the strongest defense mechanisms is \emph{Adversarial Training} (AT). In this paper, we aim to address two predominant problems in AT. First, there is still little consensus on how to set hyperparameters with a performance guarantee for AT research, and customized settings impede a fair comparison between different model designs in AT research. Second, the robustly trained neural networks struggle to generalize well and suffer from tremendous overfitting. This paper focuses on the primary AT framework - Projected Gradient Descent Adversarial Training (PGD-AT). We approximate the dynamic of PGD-AT by a continuous-time Stochastic Differential Equation (SDE), and show that the diffusion term of this SDE determines the robust generalization. An immediate implication of this theoretical finding is that robust generalization is positively correlated with the ratio between learning rate and batch size. We further propose a novel approach, \emph{Diffusion Enhanced Adversarial Training} (DEAT), to manipulate the diffusion term to improve robust generalization with virtually no extra computational burden. We theoretically show that DEAT obtains a tighter generalization bound than PGD-AT. Our empirical investigation is extensive and firmly attests that DEAT universally outperforms PGD-AT by a significant margin.
Accepted at KDD 2023
References in corpus (10)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Explaining and Harnessing Adversarial Examples
- Theoretically Principled Trade-off between Robustness and Accuracy
- Fast is better than free: Revisiting adversarial training
- On the Convergence and Robustness of Adversarial Training
- Width Provably Matters in Optimization for Deep Linear Neural Networks
- Adversarial Weight Perturbation Improves Generalization in Graph Neural Networks
- Exploring Model Robustness with Adaptive Networks and Improved Adversarial Training
- PlayBest: Professional Basketball Player Behavior Synthesis via Planning with Diffusion
- MLP Fusion: Towards Efficient Fine-tuning of Dense and Mixture-of-Experts Language Models