Interpretable Multi-dataset Evaluation for Named Entity Recognition
arXiv:2011.06854
Abstract
With the proliferation of models for natural language processing tasks, it is even harder to understand the differences between models and their relative merits. Simply looking at differences between holistic metrics such as accuracy, BLEU, or F1 does not tell us why or how particular methods perform differently and how diverse datasets influence the model design choices. In this paper, we present a general methodology for interpretable evaluation for the named entity recognition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the interplay between them, identifying the strengths and weaknesses of current systems. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive progress in this area: https://github.com/neulab/InterpretEval.
arXiv commentAccepted by EMNLP 2020
References in corpus (5)
- Analysis of Named Entity Recognition and Linking for Tweets
- GraphIE: A Graph-Based Framework for Information Extraction
- compare-mt: A Tool for Holistic Comparison of Language Generation Systems
- Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study
- RethinkCWS: Is Chinese Word Segmentation a Solved Task?