7 papers
Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judges
Chen Feng, Minghe Shen, Ananth Balashankar +2
Reliable certification of Large Language Models (LLMs)-verifying that failure rates are below a safety threshold-is critical yet challenging. While "LLM-as-a-Judge" offers scalabil…
Evaluating Medical LLMs by Levels of Autonomy: A Survey Moving from Benchmarks to Applications
Xiao Ye, Jacob Dineen, Zhaonan Li +11
Medical Large language models achieve strong scores on standard benchmarks; however, the transfer of those results to safe and reliable performance in clinical workflows remains a…
TALM: Dynamic Tree-Structured Multi-Agent Framework with Long-Term Memory for Scalable Code Generation
Ming-Tung Shen, Yuh-Jzer Joung
Agentic code generation requires large language models (LLMs) capable of complex context management and multi-step reasoning. Prior multi-agent frameworks attempt to address these…
CC-LEARN: Cohort-based Consistency Learning
Xiao Ye, Shaswat Shrivastava, Zhaonan Li +6
Large language models excel at many tasks but still struggle with consistent, robust reasoning. We introduce Cohort-based Consistency Learning (CC-Learn), a reinforcement learning…
QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA
Jacob Dineen, Aswin RRV, Qin Liu +8
Alignment of large language models (LLMs) with principles like helpfulness, honesty, and harmlessness typically relies on scalar rewards that obscure which objectives drive the tra…
Optimizing LLM-Based Multi-Agent System with Textual Feedback: A Case Study on Software Development
Ming Shen, Raphael Shu, Anurag Pratik +4
We have seen remarkable progress in large language models (LLMs) empowered multi-agent systems solving complex tasks necessitating cooperation among experts with diverse skills. Ho…