What Makes Position Zero Special? A Mechanistic Study of Position Zero Attention Sinks in LLMs

Runyu Peng, Ruixiao Li, Mingshu Chen +5

Transformers frequently allocate disproportionate attention to specific tokens, a phenomenon known as attention sinks. Causal large language models reliably form one at position ze…