Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension

Abstract

We present a new dataset and models for comprehending paragraphs about processes (e.g., photosynthesis), an important genre of text describing a dynamic world. The new dataset, ProPara, is the first to contain natural (rather than machine-generated) text about a changing world along with a full annotation of entity states (location and existence) during those changes (81k datapoints). The end-task, tracking the location and existence of entities through the text, is challenging because the causal effects of actions are often implicit and need to be inferred. We find that previous models that have worked well on synthetic data achieve only mediocre performance on ProPara, and introduce two new neural models that exploit alternative mechanisms for state prediction, in particular using LSTM input encoding and span prediction. The new models improve accuracy by up to 19%. The dataset and models are available to the community at http://data.allenai.org/propara.

arXiv commentIn Proc. NAACL'2018

References in corpus (10)

• ADADELTA: An Adaptive Learning Rate Method
• Teaching Machines to Read and Comprehend
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Adversarial Examples for Evaluating Reading Comprehension Systems
• Tracking the World State with Recurrent Entity Networks
• A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
• Simulating Action Dynamics with Neural Process Networks
• Simpler Context-Dependent Logical Forms via Model Projections
• What Happened? Leveraging VerbNet to Predict the Effects of Actions in Procedural Text

Cited by in corpus (9)

• DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
• Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
• Procedural Reading Comprehension with Attribute-Aware Context Flow
• Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning
• A Heterogeneous Graph with Factual, Temporal and Logical Knowledge for Question Answering Over Dynamic Contexts
• Natural Language QA Approaches using Reasoning with External Knowledge
• Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning
• Be Consistent! Improving Procedural Text Comprehension using Label Consistency
• From Discourse to Narrative: Knowledge Projection for Event Relation Extraction