activity
20212026
most citedAnalyzing Feed-Forward Blocks in Transformers through the Lens of Attention Maps

8 citations · 8 across the 8 of their papers we have counts for

collaborators
Showing 2026Show all

1 paper · 1 filter

cs.CL2026

It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention

Raito Kiya, Satoki Ohashi, Kosuke Sato +6

Large Language Models (LLMs) often exhibit "Attention Sink" (AS) and the accompanying "Massive Activations" (MAs) at the initial position of a sequence. These phenomena frequently…