Searching for an Effective Defender: Benchmarking Defense against Adversarial Word Substitution

Abstract

Recent studies have shown that deep neural networks are vulnerable to intentionally crafted adversarial examples, and various methods have been proposed to defend against adversarial word-substitution attacks for neural NLP models. However, there is a lack of systematic study on comparing different defense approaches under the same attacking setting. In this paper, we seek to fill the gap of systematic studies through comprehensive researches on understanding the behavior of neural text classifiers trained by various defense methods under representative adversarial attacks. In addition, we propose an effective method to further improve the robustness of neural text classifiers against such attacks and achieved the highest accuracy on both clean and adversarial examples on AGNEWS and IMDB datasets by a significant margin.

arXiv commentAccepted by EMNLP2021 main conference

References in corpus (12)

• Efficient Estimation of Word Representations in Vector Space
• mixup: Beyond Empirical Risk Minimization
• Universal Sentence Encoder
• FreeLB: Enhanced Adversarial Training for Natural Language Understanding
• Reevaluating Adversarial Examples in Natural Language
• Towards Robustness Against Natural Language Word Substitutions
• Robustness Verification for Transformers
• Defense against Adversarial Attacks in NLP via Dirichlet Neighborhood Ensemble
• TAVAT: Token-Aware Virtual Adversarial Training for Language Understanding
• Better Robustness by More Coverage: Adversarial Training with Mixup Augmentation for Robust Fine-tuning
• Voting based ensemble improves robustness of defensive models
• Searching for a Search Method: Benchmarking Search Algorithms for Generating NLP Adversarial Examples