Neural Language Modeling for Contextualized Temporal Graph Generation
arXiv:2010.10077
Abstract
This paper presents the first study on using large-scale pre-trained language models for automated generation of an event-level temporal graph for a document. Despite the huge success of neural pre-training methods in NLP tasks, its potential for temporal reasoning over event graphs has not been sufficiently explored. Part of the reason is the difficulty in obtaining large training corpora with human-annotated events and temporal links. We address this challenge by using existing IE/NLP tools to automatically generate a large quantity (89,000) of system-produced document-graph pairs, and propose a novel formulation of the contextualized graph generation problem as a sequence-to-sequence mapping task. These strategies enable us to leverage and fine-tune pre-trained language models on the system-induced training data for the graph generation task. Our experiments show that our approach is highly effective in generating structurally and semantically valid graphs. Further, evaluation on a challenging hand-labeled, out-domain corpus shows that our method outperforms the closest existing method by a large margin on several metrics. Code and pre-trained models are available at https://github.com/madaan/temporal-graph-gen.
NAACL 2021
References in corpus (18)
- Finding and evaluating community structure in networks
- Sequence to Sequence Learning with Neural Networks
- Decoupled Weight Decay Regularization
- Finding community structure in very large networks
- Fast algorithm for detecting community structure in networks
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- Language Models are Few-Shot Learners
- The Curious Case of Neural Text Degeneration
- ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
- GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models
- Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation
- EIGEN: Event Influence GENeration using Pre-trained Language Models
- Deep Structured Neural Network for Event Temporal Relation Extraction
- An Improved Neural Baseline for Temporal Relation Extraction
- TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions
- A Structured Learning Approach to Temporal Relation Extraction
- Automatic Event Salience Identification