8 papers
Process Supervision for Chain-of-Thought Reasoning via Monte Carlo Net Information Gain
Corentin Royer, Debarun Bhattacharjya, Gaetano Rossiello +2
Multi-step reasoning improves the capabilities of large language models (LLMs) but increases the risk of errors propagating through intermediate steps. Process reward models (PRMs)…
What Would an LLM Do? Evaluating Large Language Models for Policymaking to Alleviate Homelessness
Pierre Le Coz, Jia An Liu, Debarun Bhattacharjya +2
Large language models (LLMs) are increasingly being adopted in high-stakes domains. Their potential to encode evolving social contexts and to generate plausible scenarios position…
QueryGym: Step-by-Step Interaction with Relational Databases
Haritha Ananthakrishnan, Harsha Kokel, Kelsey Sikes +4
We introduce QueryGym, an interactive environment for building, testing, and evaluating LLM-based query planning agents. Existing frameworks often tie agents to specific query lang…
FactReasoner: A Probabilistic Approach to Long-Form Factuality Assessment for Large Language Models
Radu Marinescu, Debarun Bhattacharjya, Junkyu Lee +5
Large language models (LLMs) have achieved remarkable success in generative tasks, yet they often fall short in ensuring the factual accuracy of their outputs, thus limiting their…
SIMBA UQ: Similarity-Based Aggregation for Uncertainty Quantification in Large Language Models
Debarun Bhattacharjya, Balaji Ganesan, Junkyu Lee +4
When does a large language model (LLM) know what it does not know? Uncertainty quantification (UQ) provides measures of uncertainty, such as an estimate of the confidence in an LLM…
The Consistency Hypothesis in Uncertainty Quantification for Large Language Models
Quan Xiao, Debarun Bhattacharjya, Balaji Ganesan +5
Estimating the confidence of large language model (LLM) outputs is essential for real-world applications requiring high user trust. Black-box uncertainty quantification (UQ) method…