PreDiff-LM: Pretrained Discrete Masked Diffusion Language Modeling with Hybrid Attention

Zhengtao Yao, Runhao Li, Xupeng Chen +12

Discrete masked diffusion language models support bidirectional generation and infilling, but adapting pretrained autoregressive (AR) transformers requires reconciling causal pretr…