Attention Sinks in Diffusion Language Models

Maximo Eduardo Rulli, Simone Petruzzi, Edoardo Michielon +3

Masked Diffusion Language Models (DLMs) have recently emerged as a promising alternative to traditional Autoregressive Models (ARMs). DLMs employ transformer encoders with bidirect…