Assessing agreement on classification tasks: the kappa statistic

Abstract

Currently, computational linguists and cognitive scientists working in the area of discourse and dialogue argue that their subjective judgments are reliable using several different statistics, none of which are easily interpretable or comparable to each other. Meanwhile, researchers in content analysis have already experienced the same difficulties and come up with a solution in the kappa statistic. We discuss what is wrong with reliability measures as they are currently used for discourse and dialogue work in computational linguistics and cognitive science, and argue that we would be better off as a field adopting techniques from content analysis.

9 pages

References in corpus (1)

• An investigation into the correlation of cue phrases, unfilled pauses and the structuring of spoken discourse

Cited by in corpus (51)

• Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
• Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech
• Learning Natural Coding Conventions
• A Corpus-Based Investigation of Definite Description Use
• Linear Segmentation and Segment Significance
• Sentiment Analysis For Modern Standard Arabic And Colloquial
• Parsing Argumentation Structures in Persuasive Essays
• The evolution of argumentation mining: From models to social media and emerging tools
• Generating Extractive Summaries of Scientific Paradigms
• Online Social Deception and Its Countermeasures for Trustworthy Cyberspace: A Survey
• SANA : Sentiment Analysis on Newspapers comments in Algeria
• Canonical Correlation Forests
• Iterative Residual Rescaling: An Analysis and Generalization of LSI
• SentiPers: A Sentiment Analysis Corpus for Persian
• Centrality-as-Relevance: Support Sets and Similarity as Geometric Proximity
• Usability Inspection: Novice Crowd Inspectors versus Expert
• Statistical Parsing by Machine Learning from a Classical Arabic Treebank
• Extraction of domain-specific bilingual lexicon from comparable corpora: compositional translation and ranking
• Analysis of Stopping Active Learning based on Stabilizing Predictions
• Investigating Drivers' Head and Glance Correspondence
• On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation
• Building a Decision Tree Model for Academic Advising Affairs Based on the Algorithm C 4-5
• Twitter and Polls: Analyzing and estimating political orientation of Twitter users in India General #Elections2014
• Idioms-Proverbs Lexicon for Modern Standard Arabic and Colloquial Sentiment Analysis
• Evaluation Evaluation a Monte Carlo study
• Feature Reweighting for EEG-based Motor Imagery Classification
• Evaluation of YTEX and MetaMap for clinical concept recognition
• DISCO PAL: Diachronic Spanish Sonnet Corpus with Psychological and Affective Labels
• Towards Generating Virtual Movement from Textual Instructions A Case Study in Quality Assessment
• Cross-lingual Offensive Language Identification for Low Resource Languages: The Case of Marathi
• Emotion Representation Mapping for Automatic Lexicon Construction (Mostly) Performs on Human Level
• Can Prosody Aid the Automatic Classification of Dialog Acts in Conversational Speech?
• A Computational Analysis of Collective Discourse
• Tagger Evaluation Given Hierarchical Tag Sets
• Towards a Benchmark of Natural Language Arguments
• Task-oriented Dialogue Systems: performance vs. quality-optima, a review
• Truth Discovery to Resolve Object Conflicts in Linked Data
• Learning Micro-Planning Rules for Preventative Expressions
• Improving ML Training Data with Gold-Standard Quality Metrics
• Computational Sociolinguistics: A Survey
• Identification of Fertile Translations in Medical Comparable Corpora: a Morpho-Compositional Approach
• Semi-Automatic Acquisition of Domain-Specific Translation Lexicons
• An Enhanced Corpus for Arabic Newspapers Comments
• Predictive modelling of football injuries
• CrowdCafe - Mobile Crowdsourcing Platform
• An Empirical Approach to Temporal Reference Resolution (journal version)
• Solver Recommendation For Transport Problems in Slabs Using Machine Learning
• A Corpus Study of Negative Imperatives in Natural Language Instructions
• Aligning Intraobserver Agreement by Transitivity
• Induction of Word and Phrase Alignments for Automatic Document Summarization
• Target Based Speech Act Classification in Political Campaign Text