Memorization Dynamics of Fill-in-the-Middle Pretraining

Tobias von Arx, Tanguy Dieudonné

Fill-in-the-middle (FIM) is a pretraining objective widely used to equip causal language models with infilling ability, yet its effect on verbatim memorization remains underexplore…