What do Models Learn from Question Answering Datasets?

Abstract

While models have reached superhuman performance on popular question answering (QA) datasets such as SQuAD, they have yet to outperform humans on the task of question answering itself. In this paper, we investigate if models are learning reading comprehension from QA datasets by evaluating BERT-based models across five datasets. We evaluate models on their generalizability to out-of-domain examples, responses to missing or incorrect data, and ability to handle question variations. We find that no single dataset is robust to all of our experiments and identify shortcomings in both datasets and evaluation methods. Following our analysis, we make recommendations for building future QA datasets that better evaluate the task of question answering through reading comprehension. We also release code to convert QA datasets to a shared format for easier experimentation at https://github.com/amazon-research/qa-dataset-converter.

Accepted at EMNLP 2020

References in corpus (6)

• Shortcut Learning in Deep Neural Networks
• Learning and Evaluating General Linguistic Intelligence
• Data Augmentation for BERT Fine-Tuning in Open-Domain Question Answering
• ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading Comprehension
• Benchmarking Robustness of Machine Reading Comprehension Models
• Look at the First Sentence: Position Bias in Question Answering

Cited by in corpus (8)

• KLUE: Korean Language Understanding Evaluation
• The Effect of Natural Distribution Shift on Question Answering Models
• Dataset vs Reality: Understanding Model Performance from the Perspective of Information Need
• Bridging the Gap between Language Model and Reading Comprehension: Unsupervised MRC via Self-Supervision
• True or False: Does the Deep Learning Model Learn to Detect Rumors?
• Are Multilingual BERT models robust? A Case Study on Adversarial Attacks for Multilingual Question Answering
• SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoning
• Toward Deconfounding the Influence of Entity Demographics for Question Answering Accuracy