Natural Language Inference in Context -- Investigating Contextual Reasoning over Long Texts
arXiv:2011.04864
Abstract
Natural language inference (NLI) is a fundamental NLP task, investigating the entailment relationship between two texts. Popular NLI datasets present the task at sentence-level. While adequate for testing semantic representations, they fall short for testing contextual reasoning over long texts, which is a natural part of the human inference process. We introduce ConTRoL, a new dataset for ConTextual Reasoning over Long texts. Consisting of 8,325 expert-designed "context-hypothesis" pairs with gold labels, ConTRoL is a passage-level NLI dataset with a focus on complex contextual reasoning types such as logical reasoning. It is derived from competitive selection and recruitment test (verbal reasoning test) for police recruitment, with expert level quality. Compared with previous NLI benchmarks, the materials in ConTRoL are much more challenging, involving a range of reasoning types. Empirical results show that state-of-the-art language models perform by far worse than educated humans. Our dataset can also serve as a testing-set for downstream tasks like Checking Factual Correctness of Summaries.
References in corpus (10)
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Longformer: The Long-Document Transformer
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Assessing BERT's Syntactic Abilities
- WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- Multi-hop Question Answering via Reasoning Chains
- Abductive Commonsense Reasoning
- Dialogue Natural Language Inference
- Multi-Step Inference for Reasoning Over Paragraphs