5 citations · 9 across the 4 of their papers we have counts for
4 papers
Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds
John P. Lalor, Hao Wu, Hong Yu
Incorporating Item Response Theory (IRT) into NLP tasks can provide valuable information about model performance and behavior. Traditionally, IRT models are learned using human res…
Soft Label Memorization-Generalization for Natural Language Inference
John P. Lalor, Hao Wu, Hong Yu
Often when multiple labels are obtained for a training example it is assumed that there is an element of noise that must be accounted for. It has been shown that this disagreement…
Understanding Deep Learning Performance through an Examination of Test Set Difficulty: A Psychometric Case Study
John P. Lalor, Hao Wu, Tsendsuren Munkhdalai +1
Interpreting the performance of deep learning models beyond test set accuracy is challenging. Characteristics of individual data points are often not considered during evaluation,…
Building an Evaluation Scale using Item Response Theory
John P. Lalor, Hao Wu, Hong Yu
Evaluation of NLP methods requires testing against a previously vetted gold-standard test set and reporting standard metrics (accuracy/precision/recall/F1). The current assumption…