Building an Evaluation Scale using Item Response Theory

John P. Lalor, Hao Wu, Hong Yu

Evaluation of NLP methods requires testing against a previously vetted gold-standard test set and reporting standard metrics (accuracy/precision/recall/F1). The current assumption…