8 papers
Agent Lifecycle Toolkit (ALTK): Reusable Middleware Components for Robust AI Agents
Zidane Wright, Jason Tsay, Anupama Murthi +8
As AI agents move from demos into enterprise deployments, their failure modes become consequential: a misinterpreted tool argument can corrupt production data, a silent reasoning e…
How Good Are LLMs at Processing Tool Outputs?
Kiran Kate, Yara Rizk, Poulami Ghosh +4
Most realistic task automation problems require large language models (LLMs) to call tools, which often return complex JSON responses. These responses must be further processed to…
Live API-Bench: 2500+ Live APIs for Testing Multi-Step Tool Calling
Benjamin Elder, Anupama Murthi, Jungkoo Kang +4
Large language models (LLMs) increasingly rely on external tools and APIs to execute complex tasks specified in natural language. Evaluating such tool calling capabilities in reali…
When Agents go Astray: Course-Correcting SWE Agents with PRMs
Shubham Gandhi, Jason Tsay, Jatin Ganhotra +2
Large Language Model (LLM) agents are increasingly deployed for complex, multi-step software engineering (SWE) tasks. However, their trajectories often contain costly inefficiencie…
Repairing Tool Calls Using Post-tool Execution Reflection and RAG
Jason Tsay, Zidane Wright, Gaodan Fang +3
Agentic systems interact with external systems by calling tools such as Python functions, REST API endpoints, or command line tools such as kubectl in Kubernetes. These tool calls…
NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls
Kinjal Basu, Ibrahim Abdelaziz, Kiran Kate +10
The resurgence of autonomous agents built using large language models (LLMs) to solve complex real-world tasks has brought increased focus on LLMs' fundamental ability of tool or f…