873 citations · 877 across the 10 of their papers we have counts for
10 papers
The Poisoned Chalice of LLM Evaluation Report
Jonathan Katzy, Ali Al-Kaswan, Razvan Mihai Popescu +1
Large language models are increasingly used to evaluate and support software engineering tasks, yet the validity of these evaluations is often undermined by uncertainty about wheth…
Evaluating Non-English Developer Support in Machine Learning for Software Engineering
Jonathan Katzy, Yongcheng Huang, Gopal-Raj Panchu +5
Large Language Models are increasingly used in software engineering, but both code generation and its evaluation remain predominantly English-centric. This leaves a major gap in ou…
Automated Attention Pattern Discovery at Scale in Large Language Models
Jonathan Katzy, Razvan-Mihai Popescu, Erik Mekkes +2
Large language models have found success by scaling up capabilities to work in general settings. The same can unfortunately not be said for interpretability methods. The current tr…
AST-PAC: AST-guided Membership Inference for Code
Roham Koohestani, Ali Al-Kaswan, Jonathan Katzy +1
Code Large Language Models are frequently trained on massive datasets containing restrictively licensed source code. This creates urgent data governance and copyright challenges. M…
A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metrics
Jonathan Katzy, Yongcheng Huang, Gopal-Raj Panchu +5
Large Language Models are essential coding assistants, yet their training is predominantly English-centric. In this study, we evaluate the performance of code language models in no…
The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating Large Language Models
Jonathan Katzy, Razvan Mihai Popescu, Arie van Deursen +1
The recent rise in the popularity of large language models has spurred the development of extensive code datasets needed to train them. This has left limited code available for col…
An Exploratory Investigation into Code License Infringements in Large Language Model Training Datasets
Jonathan Katzy, Răzvan-Mihai Popescu, Arie van Deursen +1
Does the training of large language models potentially infringe upon code licenses? Furthermore, are there any datasets available that can be safely used for training these models…
Language Models for Code Completion: A Practical Evaluation
Maliheh Izadi, Jonathan Katzy, Tim van Dam +3
Transformer-based language models for automatic code completion have shown great promise so far, yet the evaluation of these models rarely uses real data. This study provides both…
On the Impact of Language Selection for Training and Evaluating Programming Language Models
Jonathan Katzy, Maliheh Izadi, Arie van Deursen
The recent advancements in Transformer-based Language Models have demonstrated significant potential in enhancing the multilingual capabilities of these models. The remarkable prog…
A Survey on Distributed Machine Learning
Joost Verbraeken, Matthijs Wolting, Jonathan Katzy +3
The demand for artificial intelligence has grown significantly over the last decade and this growth has been fueled by advances in machine learning techniques and the ability to le…