2 citations · 6 across the 9 of their papers we have counts for
9 papers
A Representation-Level Assessment of Bias Mitigation in Foundation Models
Svetoslav Nizhnichenkov, Rahul Nair, Elizabeth Daly +1
We investigate how successful bias mitigation reshapes the embedding space of encoder-only and decoder-only foundation models, offering an internal audit of model behaviour through…
Humble AI in the real-world: the case of algorithmic hiring
Rahul Nair, Inge Vejsbjerg, Elizabeth Daly +2
Humble AI (Knowles et al., 2023) argues for cautiousness in AI development and deployments through scepticism (accounting for limitations of statistical learning), curiosity (accou…
Paying Alignment Tax with Contrastive Learning
Buse Sibel Korkmaz, Rahul Nair, Elizabeth M. Daly +1
Current debiasing approaches often result a degradation in model capabilities such as factual accuracy and knowledge retention. Through systematic evaluation across multiple benchm…
Foundation Models at Work: Fine-Tuning for Fairness in Algorithmic Hiring
Buse Sibel Korkmaz, Rahul Nair, Elizabeth M. Daly +3
Foundation models require fine-tuning to ensure their generative outputs align with intended results for specific tasks. Automating this fine-tuning process is challenging, as it t…
Black-box Uncertainty Quantification Method for LLM-as-a-Judge
Nico Wagner, Michael Desmond, Rahul Nair +6
LLM-as-a-Judge is a widely used method for evaluating the performance of Large Language Models (LLMs) across various tasks. We address the challenge of quantifying the uncertainty…
Aligning Human and LLM Judgments: Insights from EvalAssist on Task-Specific Evaluations and AI-assisted Assessment Strategy Preferences
Zahra Ashktorab, Michael Desmond, Qian Pan +7
Evaluation of large language model (LLM) outputs requires users to make critical judgments about the best outputs across various configurations. This process is costly and takes ti…
Human-Centered Design Recommendations for LLM-as-a-Judge
Qian Pan, Zahra Ashktorab, Michael Desmond +5
Traditional reference-based metrics, such as BLEU and ROUGE, are less effective for assessing outputs from Large Language Models (LLMs) that produce highly creative or superior-qua…
Ranking Large Language Models without Ground Truth
Amit Dhurandhar, Rahul Nair, Moninder Singh +2
Evaluation and ranking of large language models (LLMs) has become an important problem with the proliferation of these models and their impact. Evaluation methods either require hu…
Explaining Knock-on Effects of Bias Mitigation
Svetoslav Nizhnichenkov, Rahul Nair, Elizabeth Daly +1
In machine learning systems, bias mitigation approaches aim to make outcomes fairer across privileged and unprivileged groups. Bias mitigation methods work in different ways and ha…