QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
arXiv:2107.12708 · doi:10.1145/3560260
Abstract
Alongside huge volumes of research on deep learning models in NLP in the recent years, there has been also much work on benchmark datasets needed to track modeling progress. Question answering and reading comprehension have been particularly prolific in this regard, with over 80 new datasets appearing in the past two years. This study is the largest survey of the field to date. We provide an overview of the various formats and domains of the current resources, highlighting the current lacunae for future work. We further discuss the current classifications of "skills" that question answering/reading comprehension systems are supposed to acquire, and propose a new taxonomy. The supplementary materials survey the current multilingual resources and monolingual resources for languages other than English, and we discuss the implications of over-focusing on English. The study is aimed at both practitioners looking for pointers to the wealth of existing data, and at researchers working on new resources.
arXiv commentPublished in ACM Comput. Surv (2022). This version differs from the final version in that section 7 ("Languages") is not in the main paper rather than the supplementary materials
References in corpus (69)
- Language Models are Few-Shot Learners
- Teaching Machines to Read and Comprehend
- Model Cards for Model Reporting
- VQA: Visual Question Answering
- Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
- On the Cross-lingual Transferability of Monolingual Representations
- Wizard of Wikipedia: Knowledge-Powered Conversational agents
- Large-scale Simple Question Answering with Memory Networks
- The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
- The Natural Language Decathlon: Multitask Learning as Question Answering
- Assessing BERT's Syntactic Abilities
- ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
- Evaluation of Text Generation: A Survey
- A Survey of Available Corpora for Building Data-Driven Dialogue Systems
- Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering
- ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning
- Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases
- Analyzing and Characterizing User Intent in Information-seeking Conversations
- DRCD: a Chinese Machine Reading Comprehension Dataset
- WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- TEQUILA: Temporal Question Answering over Knowledge Bases
- Calibrate Before Use: Improving Few-Shot Performance of Language Models
- Dataset and Neural Recurrent Sequence Labeling Model for Open-Domain Factoid Question Answering
- A Span-Extraction Dataset for Chinese Machine Reading Comprehension
- Multilingual Extractive Reading Comprehension by Runtime Machine Translation
- Embracing data abundance: BookTest Dataset for Reading Comprehension
- Look before you Hop: Conversational Question Answering over Knowledge Graphs Using Judicious Context Expansion
- A Survey of Code-switched Speech and Language Processing
- Automatic Spanish Translation of the SQuAD Dataset for Multilingual Question Answering
- KorQuAD1.0: Korean QA Dataset for Machine Reading Comprehension
- Abductive Commonsense Reasoning
- SberQuAD -- Russian Reading Comprehension Dataset: Description and Analysis
- One-shot Learning for Question-Answering in Gaokao History Challenge
- Explaining and Improving Model Behavior with k Nearest Neighbor Representations
- Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehension
- Towards Data Distillation for End-to-end Spoken Conversational Question Answering
- A Survey on Neural Machine Reading Comprehension
- Question Answering is a Format; When is it Useful?
- FQuAD: French Question Answering Dataset
- Introducing MANtIS: a novel Multi-Domain Information Seeking Dialogues Dataset
- TableQA: a Large-Scale Chinese Text-to-SQL Dataset for Table-Aware SQL Generation
- AmbigQA: Answering Ambiguous Open-domain Questions
- RuBQ: A Russian Dataset for Question Answering over Wikidata
- Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models
- When Do You Need Billions of Words of Pretraining Data?
- Adversarial TableQA: Attention Supervision for Question Answering on Tables
- MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Models
- ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading Comprehension
- Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks
- A Sentence Cloze Dataset for Chinese Machine Reading Comprehension
- LAReQA: Language-agnostic answer retrieval from a multilingual pool
- DaNetQA: a yes/no Question Answering Dataset for the Russian Language
- TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance
- TableQA: Question Answering on Tabular Data
- "Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding
- A Dataset and Baselines for Visual Question Answering on Art
- Perhaps PTLMs Should Go to School -- A Task to Assess Open Book and Closed Book QA
- A Vietnamese Dataset for Evaluating Machine Reading Comprehension
- Project PIAF: Building a Native French Question-Answering Dataset
- A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering
- HEAD-QA: A Healthcare Dataset for Complex Reasoning
- Information Seeking in the Spirit of Learning: a Dataset for Conversational Curiosity
- XOR QA: Cross-lingual Open-Retrieval Question Answering
- ArchivalQA: A Large-scale Benchmark Dataset for Open Domain Question Answering over Historical News Collections
- ReCO: A Large Scale Chinese Reading Comprehension Dataset on Opinion
- TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions
- TIMEDIAL: Temporal Commonsense Reasoning in Dialog
Cited by in corpus (6)
- A Survey on Legal Question Answering Systems
- ShortcutLens: A Visual Analytics Approach for Exploring Shortcuts in Natural Language Understanding Dataset
- Dataset vs Reality: Understanding Model Performance from the Perspective of Information Need
- Identifying Experts in Question & Answer Portals: A Case Study on Data Science Competencies in Reddit
- ConveRT for FAQ Answering
- Investigating Post-pretraining Representation Alignment for Cross-Lingual Question Answering