12 papers
Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity
Daniel Lee, Harsh Sharma, Eunkyu Park +5
MLLM-as-a-Judge is conventionally validated by agreement with human annotations, but this metric is undefined when the human pool is culturally heterogeneous. We introduce VOIR DIR…
Demystifying Multi-Agent Debate: The Role of Confidence and Diversity
Xiaochen Zhu, Caiqi Zhang, Yizhou Chi +3
Multi-agent debate (MAD) is widely used to improve large language model (LLM) performance through test-time scaling, yet recent work shows that vanilla MAD often underperforms simp…
The Role of Ambiguity in Error Prediction via Uncertainty Quantification
Ieva Raminta StaliÅ«naitÄ, James Bishop, Andreas Vlachos
The task of Error Prediction, namely predicting whether a model output is correct, is commonly tackled with Uncertainty Quantification (UQ). However, while uncertainty metrics capt…
LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generations
Caiqi Zhang, Xiaochen Zhu, Chengzu Li +2
Hallucination remains a major challenge for the safe and trustworthy deployment of large language models (LLMs) in factual content generation. Prior work has explored confidence es…
Collaborative Evaluation of Deepfake Text with Deliberation-Enhancing Dialogue Systems
Jooyoung Lee, Xiaochen Zhu, Georgi Karadzhov +3
The proliferation of generative models has presented significant challenges in distinguishing authentic human-authored content from deepfake content. Collaborative human efforts, a…
Uncertainty Quantification for Evaluating Machine Translation Bias
Ieva Raminta StaliÅ«naitÄ, Julius Cheng, Andreas Vlachos
The predictive uncertainty of machine translation (MT) models is typically used as a quality estimation proxy. In this work, we posit that apart from confidently translating when a…