565 citations · 570 across the 6 of their papers we have counts for
6 papers
The Hitchhikers Guide to Rubric Quality Understanding and Enrichment
Zhengyang Qi, Ankit Aich, Charles Dickens +7
Rubrics distill notions of expert quality and measure agent performance. However, the quality of rubrics themselves have not been systematically measured and are often left to down…
Benchmarking Agents in Insurance Underwriting Environments
Amanda Dsouza, Ramya Ramakrishnan, Charles Dickens +2
As AI agents integrate into enterprise applications, their evaluation demands benchmarks that reflect the complexity of real-world operations. Instead, existing benchmarks overemph…
Automating Benchmark Design
Amanda Dsouza, Harit Vishwakarma, Zhengyang Qi +6
The rapid progress and widespread deployment of LLMs and LLM-powered agents has outpaced our ability to evaluate them. Hand-crafted, static benchmarks are the primary tool for asse…
Evaluating Language Model Context Windows: A "Working Memory" Test and Inference-time Correction
Amanda Dsouza, Christopher Glaze, Changho Shin +1
Large language models are prominently used in real-world applications, often tasked with reasoning over large volumes of documents. An exciting development in this space is models…
Rel-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliance
Kaitlyn Zhou, Jena D. Hwang, Xiang Ren +3
The ability to communicate uncertainty, risk, and limitation is crucial for the safety of large language models. However, current evaluations of these abilities rely on simple cali…
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao +448
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabil…