2 papers
cs.AI2026
VideoGameBench: Can Vision-Language Models complete popular video games?
Alex L. Zhang, Thomas L. Griffiths, Karthik R. Narasimhan +1
Vision-language models (VLMs) have achieved strong results on coding and math benchmarks that are challenging for humans, yet their ability to perform tasks that come naturally to…
cs.CL2024
SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?
John Yang, Carlos E. Jimenez, Alex L. Zhang +10
Autonomous systems for software engineering are now capable of fixing bugs and developing features. These systems are commonly evaluated on SWE-bench (Jimenez et al., 2024a), which…