More Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question Answering

Abstract

Textual Question Answering (QA) aims to provide precise answers to user's questions in natural language using unstructured data. One of the most popular approaches to this goal is machine reading comprehension(MRC). In recent years, many novel datasets and evaluation metrics based on classical MRC tasks have been proposed for broader textual QA tasks. In this paper, we survey 47 recent textual QA benchmark datasets and propose a new taxonomy from an application point of view. In addition, We summarize 8 evaluation metrics of textual QA tasks. Finally, we discuss current trends in constructing textual QA benchmarks and suggest directions for future work.

arXiv comment18 pages, 11 figures, 6 tables

References in corpus (12)

• Language Models are Few-Shot Learners
• Teaching Machines to Read and Comprehend
• BERTScore: Evaluating Text Generation with BERT
• ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
• REALM: Retrieval-Augmented Language Model Pre-Training
• The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
• SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
• ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
• Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering
• Quasar: Datasets for Question Answering by Search and Reading
• Embracing data abundance: BookTest Dataset for Reading Comprehension
• Conversational Question Answering: A Survey