TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions
arXiv:2005.00242
Abstract
A critical part of reading is being able to understand the temporal relationships between events described in a passage of text, even when those relationships are not explicitly stated. However, current machine reading comprehension benchmarks have practically no questions that test temporal phenomena, so systems trained on these benchmarks have no capacity to answer questions such as "what happened before/after [some event]?" We introduce TORQUE, a new English reading comprehension benchmark built on 3.2k news snippets with 21k human-generated questions querying temporal relationships. Results show that RoBERTa-large achieves an exact-match score of 51% on the test set of TORQUE, about 30% behind human performance.
arXiv comment15 pages (incl. 4 pages in the appendix); accepted to EMNLP 2020
References in corpus (11)
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Know What You Don't Know: Unanswerable Questions for SQuAD
- DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
- A Multi-Axis Annotation Scheme for Event Temporal Relations
- Evaluating Models' Local Decision Boundaries via Contrast Sets
- Question Answering is a Format; When is it Useful?
- Temporal Common Sense Acquisition with Minimal Supervision
- "Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding
- Reasoning Over Paragraph Effects in Situations
- An Improved Neural Baseline for Temporal Relation Extraction
- Temporal Information Extraction by Predicting Relative Time-lines