Benchmarking Robustness of Machine Reading Comprehension Models

Abstract

Machine Reading Comprehension (MRC) is an important testbed for evaluating models' natural language understanding (NLU) ability. There has been rapid progress in this area, with new models achieving impressive performance on various benchmarks. However, existing benchmarks only evaluate models on in-domain test sets without considering their robustness under test-time perturbations or adversarial attacks. To fill this important gap, we construct AdvRACE (Adversarial RACE), a new model-agnostic benchmark for evaluating the robustness of MRC models under four different types of adversarial attacks, including our novel distractor extraction and generation attacks. We show that state-of-the-art (SOTA) models are vulnerable to all of these attacks. We conclude that there is substantial room for building more robust MRC models and our benchmark can help motivate and measure progress in this area. We release our data and code at https://github.com/NoviScl/AdvRACE .

arXiv commentACL 2021 (Findings)

References in corpus (3)

• MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
• ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading Comprehension
• Evaluating Neural Machine Comprehension Model Robustness to Noisy Inputs and Adversarial Attacks

Cited by in corpus (7)

• CharBERT: Character-aware Pre-trained Language Model
• What do Models Learn from Question Answering Datasets?
• PALRACE: Reading Comprehension Dataset with Human Data and Labeled Rationales
• Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models
• Contrastive Fine-tuning Improves Robustness for Neural Rankers
• An Understanding-Oriented Robust Machine Reading Comprehension Model
• Achieving Model Robustness through Discrete Adversarial Training