7 papers
RubricCraft: Where Solutions Diverge, Criteria Emerge
Hosein Hasanbeig, Priyanshu Gupta, Ashish Tiwari +2
Evaluating language agents on open-ended, artifact-producing tasks is difficult: model-based graders can be inconsistent, while exhaustive programmatic checks are often impractical…
Whose turn is it? Context-guided model routing for stateful agents
Anikesh Parashar, Avik Dutta, Vashu Chauhan +6
Model routing can reduce the cost of LLM applications by assigning simpler requests to cheaper models without sacrificing the quality of the response. However, most routers treat r…
Dude, Where's My State? Execution Information Requirements for Stateful Agents
Nikita Mehrotra, Ashish Tiwari, Priyanshu Gupta +1
Long-running agents must preserve information that later steps depend on. We introduce the Execution Information Requirement (EIR), a lower bound on the information that must remai…
Coding Agents are Strong Prompt Optimizers
Agamdeep Singh, Srishti Gautam, Priyanshu Gupta +3
Search-based prompt optimizers improve prompts through iterative search: they propose edits, execute fresh rollouts, score the resulting trajectories, and retain only edits that im…
Reason Wide, Not Deep: Amortizing the Reasoning Premium into Distilled Skills
Agamdeep Singh, Srishti Gautam, Priyanshu Gupta +3
Reasoning modes of language models outperform their non-reasoning counterparts on multi-step agentic tasks, but pay a 3-6x premium in output tokens on every episode -- much of it s…
Improving Language Agents through BREW: Bootstrapping expeRientially-learned Environmental knoWledge
Shashank Kirtania, Param Biyani, Priyanshu Gupta +4
Large Language Model (LLM)-based agents are increasingly capable of complex, multi-step tasks such as GUI automation, tool use, and data manipulation, yet they cannot learn from ex…
LOGIC-LM++: Multi-Step Refinement for Symbolic Formulations
Shashank Kirtania, Priyanshu Gupta, Arjun Radhakirshna
In this paper we examine the limitations of Large Language Models (LLMs) for complex reasoning tasks. Although recent works have started to employ formal languages as an intermedia…