61 citations · 225 across the 36 of their papers we have counts for
36 papers
Characterizing Feedback Statements in Machine Learning Jupyter Notebooks
Arumoy Shome, Lu\is Cruz, Diomidis Spinellis +1
Machine learning development in Jupyter notebooks is iterative and feedback-driven. Practitioners author statements that reveal information about program execution and use it to de…
"It Comes in Notebooks": Changes and Challenges when Operationalizing ML Prototypes
Arumoy Shome, Luís Cruz, Diomidis Spinellis +1
Machine learning practitioners commonly prototype models in computational notebooks before transitioning them to automated production systems. Despite its prevalence, the concrete…
Evaluating Non-English Developer Support in Machine Learning for Software Engineering
Jonathan Katzy, Yongcheng Huang, Gopal-Raj Panchu +5
Large Language Models are increasingly used in software engineering, but both code generation and its evaluation remain predominantly English-centric. This leaves a major gap in ou…
Do Agents Dream of Root Shells? Partial-Credit Evaluation of LLM Agents in Capture the Flag Challenges
Ali Al-Kaswan, Maksim Plotnikov, Maxim Hájek +3
Large Language Model (LLM) agents are increasingly proposed for autonomous cybersecurity tasks, but their capabilities in realistic offensive settings remain poorly understood. We…
Learning from Change: Predictive Models for Incident Prevention in a Regulated IT Environment
Eileen Kapel, Jan Lennartz, Luis Cruz +2
Effective IT change management is important for businesses that depend on software and services, particularly in highly regulated sectors such as finance, where operational reliabi…
TreeRanker: Fast and Model-agnostic Ranking System for Code Suggestions in IDEs
Daniele Cipollone, Egor Bogomolov, Arie van Deursen +1
Token-level code completion is one of the most critical features in modern Integrated Development Environments (IDEs). It assists developers by suggesting relevant identifiers and…
The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating Large Language Models
Jonathan Katzy, Razvan Mihai Popescu, Arie van Deursen +1
The recent rise in the popularity of large language models has spurred the development of extensive code datasets needed to train them. This has left limited code available for col…
Understanding Feedback Mechanisms in Machine Learning Jupyter Notebooks
Arumoy Shome, Luis Cruz, Diomidis Spinellis +1
The machine learning development lifecycle is characterized by iterative and exploratory processes that rely on feedback mechanisms to ensure data and model integrity. Despite the…
A Transformer-Based Approach for Smart Invocation of Automatic Code Completion
Aral de Moor, Arie van Deursen, Maliheh Izadi
Transformer-based language models are highly effective for code completion, with much research dedicated to enhancing the content of these completions. Despite their effectiveness,…
An Exploratory Investigation into Code License Infringements in Large Language Model Training Datasets
Jonathan Katzy, Răzvan-Mihai Popescu, Arie van Deursen +1
Does the training of large language models potentially infringe upon code licenses? Furthermore, are there any datasets available that can be safely used for training these models…
Language Models for Code Completion: A Practical Evaluation
Maliheh Izadi, Jonathan Katzy, Tim van Dam +3
Transformer-based language models for automatic code completion have shown great promise so far, yet the evaluation of these models rarely uses real data. This study provides both…
McUDI: Model-Centric Unsupervised Degradation Indicator for Failure Prediction AIOps Solutions
Lorena Poenaru-Olaru, Luis Cruz, Jan Rellermeyer +1
Due to the continuous change in operational data, AIOps solutions suffer from performance degradation over time. Although periodic retraining is the state-of-the-art technique to p…
Data vs. Model Machine Learning Fairness Testing: An Empirical Study
Arumoy Shome, Luis Cruz, Arie van Deursen
Although several fairness definitions and bias mitigation techniques exist in the literature, all existing solutions evaluate fairness of Machine Learning (ML) systems after the tr…
Towards Automatic Translation of Machine Learning Visual Insights to Analytical Assertions
Arumoy Shome, Luis Cruz, Arie van Deursen
We present our vision for developing an automated tool capable of translating visual properties observed in Machine Learning (ML) visualisations into Python assertions. The tool ai…
Traces of Memorisation in Large Language Models for Code
Ali Al-Kaswan, Maliheh Izadi, Arie van Deursen
Large language models have gained significant popularity because of their ability to generate human-like text and potential applications in various fields, such as Software Enginee…
Faithful Model Explanations through Energy-Constrained Conformal Counterfactuals
Patrick Altmeyer, Mojtaba Farmanbar, Arie van Deursen +1
Counterfactual explanations offer an intuitive and straightforward way to explain black-box models and offer algorithmic recourse to individuals. To address the need for plausible…
Is Your Anomaly Detector Ready for Change? Adapting AIOps Solutions to the Real World
Lorena Poenaru-Olaru, Natalia Karpova, Luis Cruz +2
Anomaly detection techniques are essential in automating the monitoring of IT systems and operations. These techniques imply that machine learning algorithms are trained on operati…
Dynamic Prediction of Delays in Software Projects using Delay Patterns and Bayesian Modeling
Elvan Kula, Eric Greuter, Arie van Deursen +1
Modern agile software projects are subject to constant change, making it essential to re-asses overall delay risk throughout the project life cycle. Existing effort estimation mode…
On the Impact of Language Selection for Training and Evaluating Programming Language Models
Jonathan Katzy, Maliheh Izadi, Arie van Deursen
The recent advancements in Transformer-based Language Models have demonstrated significant potential in enhancing the multilingual capabilities of these models. The remarkable prog…
Endogenous Macrodynamics in Algorithmic Recourse
Patrick Altmeyer, Giovan Angela, Aleksander Buszydlik +3
Existing work on Counterfactual Explanations (CE) and Algorithmic Recourse (AR) has largely focused on single individuals in a static environment: given some estimated model, the g…
Explaining Black-Box Models through Counterfactuals
Patrick Altmeyer, Arie van Deursen, Cynthia C. S. Liem
We present CounterfactualExplanations.jl: a package for generating Counterfactual Explanations (CE) and Algorithmic Recourse (AR) for black-box models in Julia. CE explain how inpu…
Batching for Green AI -- An Exploratory Study on Inference
Tim Yarally, Luís Cruz, Daniel Feitosa +2
The batch size is an essential parameter to tune during the development of new neural networks. Amongst other quality indicators, it has a large degree of influence on the model's…
Towards Understanding Machine Learning Testing in Practise
Arumoy Shome, Luis Cruz, Arie van Deursen
Visualisations drive all aspects of the Machine Learning (ML) Development Cycle but remain a vastly untapped resource by the research community. ML testing is a highly interactive…
Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study
Tim van Dam, Maliheh Izadi, Arie van Deursen
Transformer-based pre-trained models have recently achieved great results in solving many software engineering tasks including automatic code completion which is a staple in a deve…