7 papers
Beyond Pass@k: Measuring Reliability and Security of Agentic Code Generation
Jiajun Jiang, Sharon Zheng, Natan Vidra +1
AI coding agent benchmarks rank agents with the Chen et al. (2021) pass@k estimator, but current implementations misapply it: they set n to the number of unit tests in a single sub…
DevIntent: How Much Does LLM-Generated Code Violate Developer Intent?
Susana Haing, Natan Vidra, Spurthi Setty
Code generated by LLMs can violate a developer's implicit intentions when given an ambiguous prompt, yet standard benchmarks measure only whether code passes its stated test. We in…
OrchestraBench: Evaluating Multi-Agent Orchestration Failure Modes, Recovery, and Decomposition Quality
Yidian Chen, Yingzi Gu, Natan Vidra +2
Multi-agent orchestration frameworks are moving from demos to production, yet benchmarks typically report task accuracy without diagnosing why a pipeline failed, where a cascade be…
Posture and Sustainment Optimization Under Adversarial Uncertainty
Amelie Norris, Alyssa Lee, Natan Vidra +1
Pre-commitment posture, the assignment of military assets to theater locations before conflict scenarios resolve, is a critical and formally unsolved problem in joint operational p…
AdvPlan-Bench: Adversarial Evaluation of Structured Plan-Generation Agents
Alina Kapanova, Arun Kanhai, Natan Vidra +1
Structured plan-generation agents are often evaluated as if a plan has quality in isolation, yet many realistic planning tasks require asking how a candidate behaves when another a…
MetaRoute-Bench: Evaluating Meta-Decision Policies for Agentic Workflow Routing
Natan Vidra, Alina Kapanova, Arun Kanhai +1
Agentic systems must repeatedly decide whether to answer directly, decompose a task, invoke a tool, execute code, delegate to a specialist, verify an intermediate result, or recove…