11 papers
LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models
Daniele Cipollone, Sergey Titov, Maliheh Izadi +2
Large software projects often depend on older versions of libraries, even as APIs continue to evolve across releases. This creates a challenge for LLMs: they must maintain knowledg…
Evaluating Non-English Developer Support in Machine Learning for Software Engineering
Jonathan Katzy, Yongcheng Huang, Gopal-Raj Panchu +5
Large Language Models are increasingly used in software engineering, but both code generation and its evaluation remain predominantly English-centric. This leaves a major gap in ou…
Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs
Ali Al-Kaswan, Claudio Spiess, Prem Devanbu +2
Large language models are increasingly used for code generation and debugging, but their outputs can still contain bugs, that originate from training data. Distinguishing whether a…
The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating Large Language Models
Jonathan Katzy, Razvan Mihai Popescu, Arie van Deursen +1
The recent rise in the popularity of large language models has spurred the development of extensive code datasets needed to train them. This has left limited code available for col…
Evaluating Large Language Models for Functional and Maintainable Code in Industrial Settings: A Case Study at ASML
Yash Mundhra, Max Valk, Maliheh Izadi
Large language models have shown impressive performance in various domains, including code generation across diverse open-source domains. However, their applicability in proprietar…
TreeRanker: Fast and Model-agnostic Ranking System for Code Suggestions in IDEs
Daniele Cipollone, Egor Bogomolov, Arie van Deursen +1
Token-level code completion is one of the most critical features in modern Integrated Development Environments (IDEs). It assists developers by suggesting relevant identifiers and…