6 citations · 11 across the 6 of their papers we have counts for
4 papers · 1 filter
E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch
Hantian Ding, Chloe Bi, Jiacheng Zhu +5
Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories. However, evaluating repository-s…
ProgramBench: Can Language Models Rebuild Programs From Scratch?
John Yang, Kilian Lieret, Jeffrey Ma +9
Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over exten…
Disproving Program Equivalence with LLMs
Miltiadis Allamanis, Pengcheng Yin
To evaluate large language models (LLMs) for code, research has used manually created unit test-based benchmarks. However, these tests are often inadequate, missing corner cases an…
Unsupervised Evaluation of Code LLMs with Round-Trip Correctness
Miltiadis Allamanis, Sheena Panthaplackel, Pengcheng Yin
To evaluate code large language models (LLMs), research has relied on a few small manually curated benchmarks, such as HumanEval and MBPP, which represent a narrow part of the real…