Reasoning Over Paragraph Effects in Situations

Abstract

A key component of successfully reading a passage of text is the ability to apply knowledge gained from the passage to a new situation. In order to facilitate progress on this kind of reading, we present ROPES, a challenging benchmark for reading comprehension targeting Reasoning Over Paragraph Effects in Situations. We target expository language describing causes and effects (e.g., "animal pollinators increase efficiency of fertilization in flowers"), as they have clear implications for new situations. A system is presented a background passage containing at least one of these relations, a novel situation that uses this background, and questions that require reasoning about effects of the relationships in the background passage in the context of the situation. We collect background passages from science textbooks and Wikipedia that contain such phenomena, and ask crowd workers to author situations, questions, and answers, resulting in a 14,322 question dataset. We analyze the challenges of this task and evaluate the performance of state-of-the-art reading comprehension models. The best model performs only slightly better than randomly guessing an answer of the correct type, at 61.6% F1, well below the human performance of 89.0%.

References in corpus (3)

• A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
• RACE: Large-scale ReAding Comprehension Dataset From Examinations
• Interpretation of Natural Language Rules in Conversational Machine Reading

Cited by in corpus (5)

• Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models
• ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading Comprehension
• Procedural Reading Comprehension with Attribute-Aware Context Flow
• TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions
• CogME: A Cognition-Inspired Multi-Dimensional Evaluation Metric for Story Understanding