11 papers
Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?
Thibaud Gloaguen, Niels Mündler, Mark Müller +2
A widespread practice in software development is to tailor coding agents to repositories using context files, such as AGENTS.md. Although this practice is strongly encouraged by ag…
CodeTaste: Can LLMs Generate Human-Level Code Refactorings?
Alex Thillen, Niels Mündler, Veselin Raychev +1
LLM coding agents can generate working code, but their solutions often accumulate complexity, duplication, and architectural debt. Human developers address such issues through refa…
AutoBaxBuilder: Bootstrapping Code Security Benchmarking
Tobias von Arx, Niels Mündler, Mark Vero +2
As large language models (LLMs) see wide adoption in software engineering, the reliable assessment of the correctness and security of LLM-generated code is crucial. Notably, prior…
Coding Agents Don't Know When to Act
Thibaud Gloaguen, Niels Mündler, Mark Müller +2
Coding agents are increasingly deployed to autonomously maintain software, including to resolve user-reported issues: a bug report comes in and the agent creates a patch to address…
SecPI: Secure Code Generation with Reasoning Models via Security Reasoning Internalization
Hao Wang, Niels Mündler, Mark Vero +3
Reasoning language models (RLMs) are increasingly used in programming. Yet, even state-of-the-art RLMs frequently introduce critical security vulnerabilities in generated code. Pri…
Humanity's Last Exam
Long Phan, Alice Gatti, Ziwen Han +1144
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achi…