84 citations · 190 across the 34 of their papers we have counts for
35 papers
LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models
Daniele Cipollone, Sergey Titov, Maliheh Izadi +2
Large software projects often depend on older versions of libraries, even as APIs continue to evolve across releases. This creates a challenge for LLMs: they must maintain knowledg…
Evaluating Non-English Developer Support in Machine Learning for Software Engineering
Jonathan Katzy, Yongcheng Huang, Gopal-Raj Panchu +5
Large Language Models are increasingly used in software engineering, but both code generation and its evaluation remain predominantly English-centric. This leaves a major gap in ou…
Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs
Ali Al-Kaswan, Claudio Spiess, Prem Devanbu +2
Large language models are increasingly used for code generation and debugging, but their outputs can still contain bugs, that originate from training data. Distinguishing whether a…
Evaluating Large Language Models for Functional and Maintainable Code in Industrial Settings: A Case Study at ASML
Yash Mundhra, Max Valk, Maliheh Izadi
Large language models have shown impressive performance in various domains, including code generation across diverse open-source domains. However, their applicability in proprietar…
TreeRanker: Fast and Model-agnostic Ranking System for Code Suggestions in IDEs
Daniele Cipollone, Egor Bogomolov, Arie van Deursen +1
Token-level code completion is one of the most critical features in modern Integrated Development Environments (IDEs). It assists developers by suggesting relevant identifiers and…
A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metrics
Jonathan Katzy, Yongcheng Huang, Gopal-Raj Panchu +5
Large Language Models are essential coding assistants, yet their training is predominantly English-centric. In this study, we evaluate the performance of code language models in no…
Code Red! On the Harmfulness of Applying Off-the-shelf Large Language Models to Programming Tasks
Ali Al-Kaswan, Sebastian Deatc, Begüm Koç +2
Nowadays, developers increasingly rely on solutions powered by Large Language Models (LLM) to assist them with their coding tasks. This makes it crucial to align these tools with h…
A Multi-agent Onboarding Assistant based on Large Language Models, Retrieval Augmented Generation, and Chain-of-Thought
Andrei Cristian Ionescu, Sergey Titov, Maliheh Izadi
Effective onboarding in software engineering is crucial but difficult due to the fast-paced evolution of technologies. Traditional methods, like exploration and workshops, are cost…
How Much Do Code Language Models Remember? An Investigation on Data Extraction Attacks before and after Fine-tuning
Fabio Salerno, Ali Al-Kaswan, Maliheh Izadi
Code language models, while widely popular, are often trained on unsanitized source code gathered from across the Internet. Previous work revealed that pre-trained models can remem…
Rethinking IDE Customization for Enhanced HAX: A Hyperdimensional Perspective
Roham Koohestani, Maliheh Izadi
As Integrated Development Environments (IDEs) increasingly integrate Artificial Intelligence, Software Engineering faces both benefits like productivity gains and challenges like m…
The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating Large Language Models
Jonathan Katzy, Razvan Mihai Popescu, Arie van Deursen +1
The recent rise in the popularity of large language models has spurred the development of extensive code datasets needed to train them. This has left limited code available for col…
Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Tests
Amirhossein Deljouyi, Roham Koohestani, Maliheh Izadi +1
Automated unit test generators, particularly search-based software testing tools like EvoSuite, are capable of generating tests with high coverage. Although these generators allevi…
Long Code Arena: a Set of Benchmarks for Long-Context Code Models
Egor Bogomolov, Aleksandra Eliseeva, Timur Galimzyanov +8
Nowadays, the fields of code and natural language processing are evolving rapidly. In particular, models become better at processing long context windows - supported context sizes…
A Transformer-Based Approach for Smart Invocation of Automatic Code Completion
Aral de Moor, Arie van Deursen, Maliheh Izadi
Transformer-based language models are highly effective for code completion, with much research dedicated to enhancing the content of these completions. Despite their effectiveness,…
An Exploratory Investigation into Code License Infringements in Large Language Model Training Datasets
Jonathan Katzy, Răzvan-Mihai Popescu, Arie van Deursen +1
Does the training of large language models potentially infringe upon code licenses? Furthermore, are there any datasets available that can be safely used for training these models…
Investigating the Performance of Language Models for Completing Code in Functional Programming Languages: a Haskell Case Study
Tim van Dam, Frank van der Heijden, Philippe de Bekker +3
Language model-based code completion models have quickly grown in use, helping thousands of developers write code in many different programming languages. However, research on code…
Language Models for Code Completion: A Practical Evaluation
Maliheh Izadi, Jonathan Katzy, Tim van Dam +3
Transformer-based language models for automatic code completion have shown great promise so far, yet the evaluation of these models rarely uses real data. This study provides both…
In-IDE Human-AI Experience in the Era of Large Language Models; A Literature Review
Agnia Sergeyuk, Sergey Titov, Maliheh Izadi
Integrated Development Environments (IDEs) have become central to modern software development, especially with the integration of Artificial Intelligence (AI) to enhance programmin…
Traces of Memorisation in Large Language Models for Code
Ali Al-Kaswan, Maliheh Izadi, Arie van Deursen
Large language models have gained significant popularity because of their ability to generate human-like text and potential applications in various fields, such as Software Enginee…
On the Impact of Language Selection for Training and Evaluating Programming Language Models
Jonathan Katzy, Maliheh Izadi, Arie van Deursen
The recent advancements in Transformer-based Language Models have demonstrated significant potential in enhancing the multilingual capabilities of these models. The remarkable prog…
Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study
Tim van Dam, Maliheh Izadi, Arie van Deursen
Transformer-based pre-trained models have recently achieved great results in solving many software engineering tasks including automatic code completion which is a staple in a deve…
STACC: Code Comment Classification using SentenceTransformers
Ali Al-Kaswan, Maliheh Izadi, Arie van Deursen
Code comments are a key resource for information about software artefacts. Depending on the use case, only some types of comments are useful. Thus, automatic approaches to classify…
The (ab)use of Open Source Code to Train Large Language Models
Ali Al-Kaswan, Maliheh Izadi
In recent years, Large Language Models (LLMs) have gained significant popularity due to their ability to generate human-like text and their potential applications in various fields…
Targeted Attack on GPT-Neo for the SATML Language Model Data Extraction Challenge
Ali Al-Kaswan, Maliheh Izadi, Arie van Deursen
Previous work has shown that Large Language Models are susceptible to so-called data extraction attacks. This allows an attacker to extract a sample that was contained in the train…