5 papers
Improved Confidence Estimates for Black-Box Large Language Models
Sokhna Diarra Mbacke, Mouloud Belbahri, Gabriel Loaiza-Ganem
Uncertainty quantification (UQ) is essential for the safe deployment of large language models (LLMs). Existing methods, from verbalized confidence to ones requiring multiple genera…
Beyond Procedure: Substantive Fairness in Conformal Prediction
Pengqi Liu, Zijun Yu, Mouloud Belbahri +3
Conformal prediction (CP) offers distribution-free uncertainty quantification for machine learning models, yet its interplay with fairness in downstream decision-making remains und…
On the Burden of Achieving Fairness in Conformal Prediction
Ziang Gao, Pengqi Liu, Archer Yi Yang +3
Conformal prediction is often calibrated with a single pooled threshold, but this can hide cross-group heterogeneity in score distributions and distort group-wise coverage. We stud…
Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systems
Kin Kwan Leung, Mouloud Belbahri, Yi Sui +4
Retrieval-augmented generation (RAG) is a prevalent approach for building LLM-based question-answering systems that can take advantage of external knowledge databases. Due to the c…
Conformal Prediction Sets Can Cause Disparate Impact
Jesse C. Cresswell, Bhargava Kumar, Yi Sui +1
Conformal prediction is a statistically rigorous method for quantifying uncertainty in models by having them output sets of predictions, with larger sets indicating more uncertaint…