Better Robustness by More Coverage: Adversarial Training with Mixup Augmentation for Robust Fine-tuning

Abstract

Pretrained language models (PLMs) perform poorly under adversarial attacks. To improve the adversarial robustness, adversarial data augmentation (ADA) has been widely adopted to cover more search space of adversarial attacks by adding textual adversarial examples during training. However, the number of adversarial examples for text augmentation is still extremely insufficient due to the exponentially large attack search space. In this work, we propose a simple and effective method to cover a much larger proportion of the attack search space, called Adversarial and Mixup Data Augmentation (AMDA). Specifically, AMDA linearly interpolates the representations of pairs of training samples to form new virtual samples, which are more abundant and diverse than the discrete text adversarial examples in conventional ADA. Moreover, to fairly evaluate the robustness of different models, we adopt a challenging evaluation setup, which generates a new set of adversarial examples targeting each model. In text classification experiments of BERT and RoBERTa, AMDA achieves significant robustness gains under two strong adversarial attacks and alleviates the performance degradation of ADA on the clean data. Our code is available at: https://github.com/thunlp/MixADA .

arXiv commentACL 2021 (Findings)

References in corpus (13)

• BAE: BERT-based Adversarial Examples for Text Classification
• FreeLB: Enhanced Adversarial Training for Natural Language Understanding
• Adversarial Examples for Evaluating Reading Comprehension Systems
• Augmenting Data with Mixup for Sentence Classification: An Empirical Study
• It's Morphin' Time! Combating Linguistic Discrimination with Inflectional Perturbations
• CharBERT: Character-aware Pre-trained Language Model
• Evaluating Models' Local Decision Boundaries via Contrast Sets
• Defense against Adversarial Attacks in NLP via Dirichlet Neighborhood Ensemble
• TAVAT: Token-Aware Virtual Adversarial Training for Language Understanding
• Robust Machine Comprehension Models via Adversarial Training
• On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models
• AdvAug: Robust Adversarial Augmentation for Neural Machine Translation
• Sequence-Level Mixed Sample Data Augmentation

Cited by in corpus (5)

• Data Augmentation Approaches in Natural Language Processing: A Survey
• Pre-Trained Models: Past, Present and Future
• Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models
• Searching for an Effective Defender: Benchmarking Defense against Adversarial Word Substitution
• Achieving Model Robustness through Discrete Adversarial Training