Does SWE-Bench-Verified Test Agent Ability or Model Memory?

Thanosan Prathifkumar, Noble Saji Mathews, Meiyappan Nagappan

SWE-Bench-Verified, a dataset comprising 500 issues, serves as a de facto benchmark for evaluating various large language models (LLMs) on their ability to resolve GitHub issues. B…