6 citations · 18 across the 15 of their papers we have counts for
13 papers · 1 filter
Carefully Considering Culture: Analyzing LLM Alignment in Single- and Multi-Cultural Settings using Cultural Consensus Theory
Krishna Pothugunta, John P. Lalor
Recent work in NLP has probed large language models for their understanding of cultural norms across countries. However, this work typically considers distributional patterns, igno…
Bridging the LLM Accessibility Divide? Performance, Fairness, and Cost of Closed versus Open LLMs for Automated Essay Scoring
Kezia Oketch, John P. Lalor, Yi Yang +1
Closed large language models (LLMs) such as GPT-4 have set state-of-the-art results across a number of NLP tasks and have become central to NLP and machine learning (ML)-driven sol…
A Psychology-based Unified Dynamic Framework for Curriculum Learning
Guangyu Meng, Qingkai Zeng, John P. Lalor +1
Directly learning from examples of varying difficulty levels is often challenging for both humans and machine learning models. A more effective strategy involves exposing learners…
H-COAL: Human Correction of AI-Generated Labels for Biomedical Named Entity Recognition
Xiaojing Duan, John P. Lalor
With the rapid advancement of machine learning models for NLP tasks, collecting high-fidelity labels from AI models is a realistic possibility. Firms now make AI available to custo…
Bias A-head? Analyzing Bias in Transformer-Based Language Model Attention Heads
Yi Yang, Hanyu Duan, Ahmed Abbasi +2
Transformer-based pretrained large language models (PLM) such as BERT and GPT have achieved remarkable success in NLP tasks. However, PLMs are prone to encoding stereotypical biase…
From Stars to Insights: Exploration and Implementation of Unified Sentiment Analysis with Distant Supervision
Wenchang Li, John P. Lalor, Yixing Chen +1
Sentiment analysis is integral to understanding the voice of the customer and informing businesses' strategic decisions. Conventional sentiment analysis involves three separate tas…
py-irt: A Scalable Item Response Theory Library for Python
John P. Lalor, Pedro Rodriguez
py-irt is a Python library for fitting Bayesian Item Response Theory (IRT) models. py-irt estimates latent traits of subjects and items, making it appropriate for use in IRT tasks…
Dynamic Data Selection for Curriculum Learning via Ability Estimation
John P. Lalor, Hong Yu
Curriculum learning methods typically rely on heuristics to estimate the difficulty of training examples or the ability of the model. In this work, we propose replacing difficulty…
Efficient Semi-Supervised Learning for Natural Language Understanding by Optimizing Diversity
Eunah Cho, He Xie, John P. Lalor +2
Expanding new functionalities efficiently is an ongoing challenge for single-turn task-oriented dialogue systems. In this work, we explore functionality-specific semi-supervised le…
Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds
John P. Lalor, Hao Wu, Hong Yu
Incorporating Item Response Theory (IRT) into NLP tasks can provide valuable information about model performance and behavior. Traditionally, IRT models are learned using human res…
Soft Label Memorization-Generalization for Natural Language Inference
John P. Lalor, Hao Wu, Hong Yu
Often when multiple labels are obtained for a training example it is assumed that there is an element of noise that must be accounted for. It has been shown that this disagreement…
Understanding Deep Learning Performance through an Examination of Test Set Difficulty: A Psychometric Case Study
John P. Lalor, Hao Wu, Tsendsuren Munkhdalai +1
Interpreting the performance of deep learning models beyond test set accuracy is challenging. Characteristics of individual data points are often not considered during evaluation,…
Building an Evaluation Scale using Item Response Theory
John P. Lalor, Hao Wu, Hong Yu
Evaluation of NLP methods requires testing against a previously vetted gold-standard test set and reporting standard metrics (accuracy/precision/recall/F1). The current assumption…