ARCH: Efficient Adversarial Regularized Training with Caching
arXiv:2109.07048
Abstract
Adversarial regularization can improve model generalization in many natural language processing tasks. However, conventional approaches are computationally expensive since they need to generate a perturbation for each sample in each epoch. We propose a new adversarial regularization method ARCH (adversarial regularization with caching), where perturbations are generated and cached once every several epochs. As caching all the perturbations imposes memory usage concerns, we adopt a K-nearest neighbors-based strategy to tackle this issue. The strategy only requires caching a small amount of perturbations, without introducing additional training time. We evaluate our proposed method on a set of neural machine translation and natural language understanding tasks. We observe that ARCH significantly eases the computational burden (saves up to 70% of computational time in comparison with conventional approaches). More surprisingly, by reducing the variance of stochastic gradients, ARCH produces a notably better (in most of the tasks) or comparable model generalization. Our code is available at https://github.com/SimiaoZuo/Caching-Adv.
arXiv commentEMNLP 2021 (findings)
References in corpus (8)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Convolutional Sequence to Sequence Learning
- DeBERTa: Decoding-enhanced BERT with Disentangled Attention
- FreeLB: Enhanced Adversarial Training for Natural Language Understanding
- There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average
- Adversarial Training for Large Neural Language Models
- Improving Neural Language Modeling via Adversarial Training
- The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalization