3 papers
cs.AI2026
Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents
Akshay Manglik, Apaar Shanker, Kaustubh Deshpande +6
Diagnosing failures in LLM agents remains largely manual. Practitioners inspect a small subset of execution traces, form ad-hoc hypotheses, and iterate. This process misses pattern…
cs.CY2025
FORTRESS: Frontier Risk Evaluation for National Security and Public Safety
Christina Q. Knight, Kaustubh Deshpande, Ved Sirdeshmukh +4
The rapid advancement of large language models (LLMs) introduces dual-use capabilities that could both threaten and bolster national security and public safety (NSPS). Models imple…
cs.CL2025
MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs
Ved Sirdeshmukh, Kaustubh Deshpande, Johannes Mols +7
We present MultiChallenge, a pioneering benchmark evaluating large language models (LLMs) on conducting multi-turn conversations with human users, a crucial yet underexamined capab…