A Qualitative Comparison of CoQA, SQuAD 2.0 and QuAC
arXiv:1809.10735
Abstract
We compare three new datasets for question answering: SQuAD 2.0, QuAC, and CoQA, along several of their new features: (1) unanswerable questions, (2) multi-turn interactions, and (3) abstractive answers. We show that the datasets provide complementary coverage of the first two aspects, but weak coverage of the third. Because of the datasets' structural similarity, a single extractive model can be easily adapted to any of the datasets and we show improved baseline results on both SQuAD 2.0 and CoQA. Despite the similarity, models trained on one dataset are ineffective on another dataset, but we find moderate performance improvement through pretraining. To encourage cross-evaluation, we release code for conversion between datasets at https://github.com/my89/co-squac .
arXiv commentCamera Ready Presented at NAACL '19
References in corpus (8)
- Teaching Machines to Read and Comprehend
- Bidirectional Attention Flow for Machine Comprehension
- VQA: Visual Question Answering
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- A large annotated corpus for learning natural language inference
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
- NewsQA: A Machine Comprehension Dataset
- Reinforced Mnemonic Reader for Machine Reading Comprehension
Cited by in corpus (4)
- SDNet: Contextualized Attention-based Deep Network for Conversational Question Answering
- GraphFlow: Exploiting Conversation Flow with Graph Neural Networks for Conversational Machine Comprehension
- FlowDelta: Modeling Flow Information Gain in Reasoning for Conversational Machine Comprehension
- Reinforced Dynamic Reasoning for Conversational Question Generation