SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments

Avi Arora, Jinu Jang, Roshanak Zilouchian Moghaddam

Modern Large Language Model (LLM) agents promise end to end assistance with real-world software tasks, yet existing benchmarks evaluate LLM agents almost exclusively in pre-baked e…