2 papers
cs.LG2025
Normalization in Attention Dynamics
Nikita Karagodin, Shu Ge, Yury Polyanskiy +1
We study the effect of normalization schemes on token representations in deep transformers. Modeling their evolution as interacting particles on the sphere, we show that normalizat…
cs.LG2024
Clustering in Causal Attention Masking
Nikita Karagodin, Yury Polyanskiy, Philippe Rigollet
This work presents a modification of the self-attention dynamics proposed by Geshkovski et al. (arXiv:2312.10794) to better reflect the practically relevant, causally masked attent…