6 papers
The Poisoned Chalice of LLM Evaluation Report
Jonathan Katzy, Ali Al-Kaswan, Razvan Mihai Popescu +1
Large language models are increasingly used to evaluate and support software engineering tasks, yet the validity of these evaluations is often undermined by uncertainty about wheth…
Evaluating Non-English Developer Support in Machine Learning for Software Engineering
Jonathan Katzy, Yongcheng Huang, Gopal-Raj Panchu +5
Large Language Models are increasingly used in software engineering, but both code generation and its evaluation remain predominantly English-centric. This leaves a major gap in ou…
Automated Attention Pattern Discovery at Scale in Large Language Models
Jonathan Katzy, Razvan-Mihai Popescu, Erik Mekkes +2
Large language models have found success by scaling up capabilities to work in general settings. The same can unfortunately not be said for interpretability methods. The current tr…
AST-PAC: AST-guided Membership Inference for Code
Roham Koohestani, Ali Al-Kaswan, Jonathan Katzy +1
Code Large Language Models are frequently trained on massive datasets containing restrictively licensed source code. This creates urgent data governance and copyright challenges. M…
The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating Large Language Models
Jonathan Katzy, Razvan Mihai Popescu, Arie van Deursen +1
The recent rise in the popularity of large language models has spurred the development of extensive code datasets needed to train them. This has left limited code available for col…
A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metrics
Jonathan Katzy, Yongcheng Huang, Gopal-Raj Panchu +5
Large Language Models are essential coding assistants, yet their training is predominantly English-centric. In this study, we evaluate the performance of code language models in no…