A Multi-Axis Annotation Scheme for Event Temporal Relations
arXiv:1804.07828
Abstract
Existing temporal relation (TempRel) annotation schemes often have low inter-annotator agreements (IAA) even between experts, suggesting that the current annotation task needs a better definition. This paper proposes a new multi-axis modeling to better capture the temporal structure of events. In addition, we identify that event end-points are a major source of confusion in annotation, so we also propose to annotate TempRels based on start-points only. A pilot expert annotation using the proposed scheme shows significant improvement in IAA from the conventional 60's to 80's (Cohen's Kappa). This better-defined annotation scheme further enables the use of crowdsourcing to alleviate the labor intensity for each annotator. We hope that this work can foster more interesting studies towards event understanding.
arXiv comment[Final Version] 14 pages, accepted by ACL'18
References in corpus (1)
Cited by in corpus (25)
- Evaluating Models' Local Decision Boundaries via Contrast Sets
- "Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding
- Contextualized Word Embeddings Enhanced Event Temporal Relation Extraction for Story Understanding
- EventPlus: A Temporal Event Understanding Pipeline
- Deep Structured Neural Network for Event Temporal Relation Extraction
- Joint Constrained Learning for Event-Event Relation Extraction
- Joint Event and Temporal Relation Extraction with Shared Representations and Structured Prediction
- An Improved Neural Baseline for Temporal Relation Extraction
- TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions
- Multitask Learning for Class-Imbalanced Discourse Classification
- Effective Distant Supervision for Temporal Relation Extraction
- Temporal Reasoning on Implicit Events from Distant Supervision
- Easy, Reproducible and Quality-Controlled Data Collection with Crowdaq
- Temporal Information Extraction by Predicting Relative Time-lines
- Joint Reasoning for Temporal and Causal Relations
- Exploiting Partially Annotated Data for Temporal Relation Extraction
- Improving Event Duration Prediction via Time-aware Pre-training
- ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning
- CogCompTime: A Tool for Understanding Time in Natural Language Text
- Conditional Generation of Temporally-ordered Event Sequences
- STAGE: Tool for Automated Extraction of Semantic Time Cues to Enrich Neural Temporal Ordering Models
- Extracting Event Temporal Relations via Hyperbolic Geometry
- Partial Or Complete, That's The Question
- CALM: Continuous Adaptive Learning for Language Modeling
- Embedding time expressions for deep temporal ordering models