MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension

Abstract

We present the results of the Machine Reading for Question Answering (MRQA) 2019 shared task on evaluating the generalization capabilities of reading comprehension systems. In this task, we adapted and unified 18 distinct question answering datasets into the same format. Among them, six datasets were made available for training, six datasets were made available for development, and the final six were hidden for final evaluation. Ten teams submitted systems, which explored various ideas including data sampling, multi-task learning, adversarial training and ensembling. The best system achieved an average F1 score of 72.5 on the 12 held-out datasets, 10.7 absolute points higher than our initial baseline based on BERT.

EMNLP 2019 Workshop on Machine Reading for Question Answering

References in corpus (2)

• SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
• Learning and Evaluating General Linguistic Intelligence

Cited by in corpus (7)

• From Machine Reading Comprehension to Dialogue State Tracking: Bridging the Gap
• Benchmarking Robustness of Machine Reading Comprehension Models
• KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative Learning
• Improving Robustness and Generality of NLP Models Using Disentangled Representations
• SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval
• Robust Question Answering Through Sub-part Alignment
• Knowing More About Questions Can Help: Improving Calibration in Question Answering