Improved OOD Generalization via Adversarial Training and Pre-training
arXiv:2105.11144
Abstract
Recently, learning a model that generalizes well on out-of-distribution (OOD) data has attracted great attention in the machine learning community. In this paper, after defining OOD generalization via Wasserstein distance, we theoretically show that a model robust to input perturbation generalizes well on OOD data. Inspired by previous findings that adversarial training helps improve input-robustness, we theoretically show that adversarially trained models have converged excess risk on OOD data, and empirically verify it on both image classification and natural language understanding tasks. Besides, in the paradigm of first pre-training and then fine-tuning, we theoretically show that a pre-trained model that is more robust to input perturbation provides a better initialization for generalization on downstream OOD data. Empirically, after fine-tuning, this better-initialized model from adversarial pre-training also has better OOD generalization.
arXiv commentpublished in ICML2021
References in corpus (14)
- Decoupled Weight Decay Regularization
- Intriguing properties of neural networks
- Language Models are Few-Shot Learners
- Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
- Using Pre-Training Can Improve Model Robustness and Uncertainty
- FreeLB: Enhanced Adversarial Training for Natural Language Understanding
- Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methods
- Adversarial Training Can Hurt Generalization
- Do Adversarially Robust ImageNet Models Transfer Better?
- Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
- Minimax Statistical Learning with Wasserstein Distances
- Unadversarial Examples: Designing Objects for Robust Vision
- Inductive Bias of Gradient Descent based Adversarial Training on Separable Data
- Estimating the Brittleness of AI: Safety Integrity Levels and the Need for Testing Out-Of-Distribution Performance