6 papers
An Assessment of Human vs. Model Uncertainty in Soft-Label Learning and Calibration
Maja Pavlovic, Silviu Paun, Massimo Poesio
Central to human-aligned AI is understanding the benefits of human-elicited labels over synthetic alternatives. While human soft-labels improve calibration by capturing uncertainty…
Toward Human-AI Complementarity Across Diverse Tasks
Yuzheng Xu, Annya Dahmani, Matthew D. Blanchard +13
Human-AI complementarity, the idea that combining human and AI judgments can outperform either alone, offers a promising pathway toward robust oversight of advanced AI systems. How…
LeWiDi-2025 at NLPerspectives: Third Edition of the Learning with Disagreements Shared Task
Elisa Leonardelli, Silvia Casola, Siyao Peng +8
Many researchers have reached the conclusion that AI models should be trained to be aware of the possibility of variation and disagreement in human judgments, and evaluated as per…
Understanding Model Calibration -- A gentle introduction and visual exploration of calibration and the expected calibration error (ECE)
Maja Pavlovic
To be considered reliable, a model must be calibrated so that its confidence in each decision closely reflects its true outcome. In this blogpost we'll take a look at the most comm…
The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representation
Maja Pavlovic, Massimo Poesio
Large Language Models (LLMs) have emerged as powerful support tools across various natural language tasks and a range of application domains. Recent studies focus on exploring thei…
Understanding The Effect Of Temperature On Alignment With Human Opinions
Maja Pavlovic, Massimo Poesio
With the increasing capabilities of LLMs, recent studies focus on understanding whose opinions are represented by them and how to effectively extract aligned opinion distributions.…