5 papers · 1 filter
RubricCraft: Where Solutions Diverge, Criteria Emerge
Hosein Hasanbeig, Priyanshu Gupta, Ashish Tiwari +2
Evaluating language agents on open-ended, artifact-producing tasks is difficult: model-based graders can be inconsistent, while exhaustive programmatic checks are often impractical…
Whose turn is it? Context-guided model routing for stateful agents
Anikesh Parashar, Avik Dutta, Vashu Chauhan +6
Model routing can reduce the cost of LLM applications by assigning simpler requests to cheaper models without sacrificing the quality of the response. However, most routers treat r…
Dude, Where's My State? Execution Information Requirements for Stateful Agents
Nikita Mehrotra, Ashish Tiwari, Priyanshu Gupta +1
Long-running agents must preserve information that later steps depend on. We introduce the Execution Information Requirement (EIR), a lower bound on the information that must remai…
Coding Agents are Strong Prompt Optimizers
Agamdeep Singh, Srishti Gautam, Priyanshu Gupta +3
Search-based prompt optimizers improve prompts through iterative search: they propose edits, execute fresh rollouts, score the resulting trajectories, and retain only edits that im…
Reason Wide, Not Deep: Amortizing the Reasoning Premium into Distilled Skills
Agamdeep Singh, Srishti Gautam, Priyanshu Gupta +3
Reasoning modes of language models outperform their non-reasoning counterparts on multi-step agentic tasks, but pay a 3-6x premium in output tokens on every episode -- much of it s…