Reducing Pretraining-Generation Mismatch in Diffusion Language Models

Xiaocheng Lu, Huabin Liu, Song Guo +1

Diffusion language models (dLLMs) generate text through iterative denoising, allowing multiple tokens to be predicted in parallel. However, pretraining may mask tokens throughout a…