Robust Machine Comprehension Models via Adversarial Training

Abstract

It is shown that many published models for the Stanford Question Answering Dataset (Rajpurkar et al., 2016) lack robustness, suffering an over 50% decrease in F1 score during adversarial evaluation based on the AddSent (Jia and Liang, 2017) algorithm. It has also been shown that retraining models on data generated by AddSent has limited effect on their robustness. We propose a novel alternative adversary-generation algorithm, AddSentDiverse, that significantly increases the variance within the adversarial training data by providing effective examples that punish the model for making certain superficial assumptions. Further, in order to improve robustness to AddSent's semantic perturbations (e.g., antonyms), we jointly improve the model's semantic-relationship learning capabilities in addition to our AddSentDiverse-based adversarial training data augmentation. With these additions, we show that we can make a state-of-the-art model significantly more robust, achieving a 36.5% increase in F1 score under many different types of adversarial evaluation while maintaining performance on the regular SQuAD task.

arXiv commentNAACL 2018 (7 pages)

References in corpus (3)

• Machine Comprehension Using Match-LSTM and Answer Pointer
• Adversarial Machine Learning at Scale
• Adversarial Examples for Evaluating Reading Comprehension Systems

Cited by in corpus (10)

• Adversarial Robustness of Deep Code Comment Generation
• Better Robustness by More Coverage: Adversarial Training with Mixup Augmentation for Robust Fine-tuning
• Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models
• Attention-Guided Answer Distillation for Machine Reading Comprehension
• Adversarial Attacks and Defense on Texts: A Survey
• QAInfomax: Learning Robust Question Answering System by Mutual Information Maximization
• Counterfactual Variable Control for Robust and Interpretable Question Answering
• Robust Reading Comprehension with Linguistic Constraints via Posterior Regularization
• Improving Robustness of Malware Classifiers using Adversarial Strings Generated from Perturbed Latent Representations
• Robustness to Modification with Shared Words in Paraphrase Identification