InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective

Abstract

Large-scale language models such as BERT have achieved state-of-the-art performance across a wide range of NLP tasks. Recent studies, however, show that such BERT-based models are vulnerable facing the threats of textual adversarial attacks. We aim to address this problem from an information-theoretic perspective, and propose InfoBERT, a novel learning framework for robust fine-tuning of pre-trained language models. InfoBERT contains two mutual-information-based regularizers for model training: (i) an Information Bottleneck regularizer, which suppresses noisy mutual information between the input and the feature representation; and (ii) a Robust Feature regularizer, which increases the mutual information between local robust features and global features. We provide a principled way to theoretically analyze and improve the robustness of representation learning for language models in both standard and adversarial training. Extensive experiments demonstrate that InfoBERT achieves state-of-the-art robust accuracy over several adversarial datasets on Natural Language Inference (NLI) and Question Answering (QA) tasks. Our code is available at https://github.com/AI-secure/InfoBERT.

arXiv commentAccepted to ICLR 2021. 23 pages, 9 tables, 3 figures

References in corpus (12)

• A Simple Framework for Contrastive Learning of Visual Representations
• Language Models are Few-Shot Learners
• Learning deep representations by mutual information estimation and maximization
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• What Makes for Good Views for Contrastive Learning?
• Certified Adversarial Robustness via Randomized Smoothing
• FreeLB: Enhanced Adversarial Training for Natural Language Understanding
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• Adversarial Training for Large Neural Language Models
• A Mutual Information Maximization Perspective of Language Representation Learning
• Learning Adversarially Robust Representations via Worst-Case Mutual Information Maximization

Cited by in corpus (5)

• Improving the Adversarial Robustness of NLP Models by Information Bottleneck
• Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation
• Robust Lottery Tickets for Pre-trained Language Models
• Learning Contextualized Knowledge Structures for Commonsense Reasoning
• RefBERT: Compressing BERT by Referencing to Pre-computed Representations