3 papers
cs.LG2025
Measuring all the noises of LLM Evals
Sida Wang, Sida I. Wang
Applying well-established statistical methods effectively to LLM evals requires consideration of model correlations and prediction noises. Leading evals often ignore data noise, ha…
cs.CL2025
Structure-Aware Fill-in-the-Middle Pretraining for Code
Linyuan Gong, Alvin Cheung, Mostafa Elhoushi +1
Fill-in-the-Middle (FIM) is a common pretraining method for code LLMs, where models complete code segments given surrounding context. However, existing LLMs treat code as plain tex…
cs.CL2024
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks
Linyuan Gong, Sida Wang, Mostafa Elhoushi +1
We introduce Syntax-Aware Fill-In-the-Middle (SAFIM), a new benchmark for evaluating Large Language Models (LLMs) on the code Fill-in-the-Middle (FIM) task. This benchmark focuses…