A Multi-Axis Annotation Scheme for Event Temporal Relations
arXiv:1804.07828
Abstract
Existing temporal relation (TempRel) annotation schemes often have low inter-annotator agreements (IAA) even between experts, suggesting that the current annotation task needs a better definition. This paper proposes a new multi-axis modeling to better capture the temporal structure of events. In addition, we identify that event end-points are a major source of confusion in annotation, so we also propose to annotate TempRels based on start-points only. A pilot expert annotation using the proposed scheme shows significant improvement in IAA from the conventional 60's to 80's (Cohen's Kappa). This better-defined annotation scheme further enables the use of crowdsourcing to alleviate the labor intensity for each annotator. We hope that this work can foster more interesting studies towards event understanding.
[Final Version] 14 pages, accepted by ACL'18
References in corpus (1)
Cited by in corpus (25)
- Evaluating Models' Local Decision Boundaries via Contrast Sets
- "Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding
- Contextualized Word Embeddings Enhanced Event Temporal Relation Extraction for Story Understanding
- EventPlus: A Temporal Event Understanding Pipeline
- Deep Structured Neural Network for Event Temporal Relation Extraction
- Joint Constrained Learning for Event-Event Relation Extraction
- Joint Event and Temporal Relation Extraction with Shared Representations and Structured Prediction
- An Improved Neural Baseline for Temporal Relation Extraction
- TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions
- Multitask Learning for Class-Imbalanced Discourse Classification
- Effective Distant Supervision for Temporal Relation Extraction
- Temporal Reasoning on Implicit Events from Distant Supervision
- Easy, Reproducible and Quality-Controlled Data Collection with Crowdaq
- Temporal Information Extraction by Predicting Relative Time-lines
- Joint Reasoning for Temporal and Causal Relations
- Exploiting Partially Annotated Data for Temporal Relation Extraction
- Improving Event Duration Prediction via Time-aware Pre-training
- ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning
- CogCompTime: A Tool for Understanding Time in Natural Language Text
- Conditional Generation of Temporally-ordered Event Sequences
- STAGE: Tool for Automated Extraction of Semantic Time Cues to Enrich Neural Temporal Ordering Models
- Extracting Event Temporal Relations via Hyperbolic Geometry
- Partial Or Complete, That's The Question
- CALM: Continuous Adaptive Learning for Language Modeling
- Embedding time expressions for deep temporal ordering models