Structure-Aware Fill-in-the-Middle Pretraining for Code

Linyuan Gong, Alvin Cheung, Mostafa Elhoushi +1

Fill-in-the-Middle (FIM) is a common pretraining method for code LLMs, where models complete code segments given surrounding context. However, existing LLMs treat code as plain tex…