4 papers
InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries
Samuel J. Vincent, Daniel Calloway, Fangyi Yu +2
Legal AI systems are increasingly used to answer legal questions, yet existing benchmarks assume queries arrive fully specified. In practice, users omit facts that materially deter…
To Whom Do Language Models Align? Measuring Principal Hierarchies Under High-Stakes Competing Demands
Fangyi Yu, Nabeel Seedat, Jonathan Richard Schwarz +1
Language models deployed in high-stakes professional settings face conflicting demands from users, institutional authorities, and professional norms. How models act when these dema…
Beyond Pointwise Scores: Decomposed Criteria-Based Evaluation of LLM Responses
Fangyi Yu, Nabeel Seedat, Dasha Herrmannova +2
Evaluating long-form answers in high-stakes domains such as law or medicine remains a fundamental challenge. Standard metrics like BLEU and ROUGE fail to capture semantic correctne…
When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs
Fangyi Yu
As large language models (LLMs) grow in capability and autonomy, evaluating their outputs-especially in open-ended and complex tasks-has become a critical bottleneck. A new paradig…