8 papers
CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHR
Veronica Chatrath, Bryan Zhu, George Pu +16
Large language models perform strongly on medical knowledge benchmarks, but reliable clinical deployment requires agents to conduct defensible investigations over heterogeneous, lo…
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Varun Ursekar, Apaar Shanker, Yash Maurya +4
As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory,…
Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents
Akshay Manglik, Apaar Shanker, Kaustubh Deshpande +6
Diagnosing failures in LLM agents remains largely manual. Practitioners inspect a small subset of execution traces, form ad-hoc hypotheses, and iterate. This process misses pattern…
VeRO: A Harness for Agents to Optimize Agents
Varun Ursekar, Apaar Shanker, Veronica Chatrath +2
An important emerging application of coding agents is agent harness optimization: the iterative improvement of a target agent by editing and evaluating its code. Despite its releva…
MIDST Challenge at SaTML 2025: Membership Inference over Diffusion-models-based Synthetic Tabular data
Masoumeh Shafieinejad, Xi He, Mahshid Alinoori +6
Synthetic data is often perceived as a silver-bullet solution to data anonymization and privacy-preserving data publishing. Drawn from generative models like diffusion models, synt…
ViLBias: Detecting and Reasoning about Bias in Multimodal Content
Shaina Raza, Caesar Saleh, Azib Farooq +11
Detecting bias in multimodal news requires models that reason over text--image pairs, not just classify text. In response, we present ViLBias, a VQA-style benchmark and framework f…