8 papers
Cross Lingual Transfer in Tulu Legal Comprehension: Script-Dependent Improvement and RAG-Induced Knowledge Conflict
Sindhu Shetty, Spurthi Setty, Natan Vidra
Low-resource languages without an adequate training corpus often use a related, higher-resource language as a scaffold for comprehension. Still, there is a need to develop rigorous…
Beyond Pass@k: Measuring Reliability and Security of Agentic Code Generation
Jiajun Jiang, Sharon Zheng, Natan Vidra +1
AI coding agent benchmarks rank agents with the Chen et al. (2021) pass@k estimator, but current implementations misapply it: they set n to the number of unit tests in a single sub…
DevIntent: How Much Does LLM-Generated Code Violate Developer Intent?
Susana Haing, Natan Vidra, Spurthi Setty
Code generated by LLMs can violate a developer's implicit intentions when given an ambiguous prompt, yet standard benchmarks measure only whether code passes its stated test. We in…
OrchestraBench: Evaluating Multi-Agent Orchestration Failure Modes, Recovery, and Decomposition Quality
Yidian Chen, Yingzi Gu, Natan Vidra +2
Multi-agent orchestration frameworks are moving from demos to production, yet benchmarks typically report task accuracy without diagnosing why a pipeline failed, where a cascade be…
Posture and Sustainment Optimization Under Adversarial Uncertainty
Amelie Norris, Alyssa Lee, Natan Vidra +1
Pre-commitment posture, the assignment of military assets to theater locations before conflict scenarios resolve, is a critical and formally unsolved problem in joint operational p…
AdvPlan-Bench: Adversarial Evaluation of Structured Plan-Generation Agents
Alina Kapanova, Arun Kanhai, Natan Vidra +1
Structured plan-generation agents are often evaluated as if a plan has quality in isolation, yet many realistic planning tasks require asking how a candidate behaves when another a…