8 papers
Beyond Single-Turn Confidence: Trajectory-Adapted Uncertainty Quantification for LLM Agents
Dylan Bouchard, Mohit Singh Chauhan
Uncertainty quantification (UQ) methods for language models are typically evaluated on single-turn outputs, where uncertainty is attached to one generated answer. For LLM agents, h…
Fine-Grained Uncertainty Quantification for Long-Form Language Model Outputs: A Comparative Study
Dylan Bouchard, Mohit Singh Chauhan, Viren Bajaj +1
Uncertainty quantification has emerged as an effective approach to closed-book hallucination detection for LLMs, but existing methods are largely designed for short-form outputs an…
Functional Entropy: Predicting Functional Correctness in LLM-Generated Code with Uncertainty Quantification
Dylan Bouchard, Mohit Singh Chauhan, Zeya Ahmad +1
Large language models have shown impressive capabilities in code generation, yet they often produce functionally incorrect code. Uncertainty quantification (UQ) methods have emerge…
Bring Your Own Prompts: Use-Case-Specific Bias and Fairness Evaluation for LLMs
Dylan Bouchard
Bias and fairness risks in Large Language Models (LLMs) vary substantially across deployment contexts, yet existing approaches lack systematic guidance for selecting appropriate ev…
Is Escalation Worth It? A Decision-Theoretic Characterization of LLM Cascades
Dylan Bouchard
Model cascades, in which a cheap LLM defers to an expensive one on low-confidence queries, are widely used to navigate the cost-quality tradeoff at deployment. Existing approaches…
UQLM: A Python Package for Uncertainty Quantification in Large Language Models
Dylan Bouchard, Mohit Singh Chauhan, David Skarbrevik +3
Hallucinations, defined as instances where Large Language Models (LLMs) generate false or misleading content, pose a significant challenge that impacts the safety and trust of down…