Improved Large Language Diffusion Models

Shen Nie, Qiyang Min, Shaoxuan Xu +7

Modern large language models are predominantly trained with autoregressive factorization and causal attention. We present \emph{iLLaDA}, an 8B masked diffusion language model train…