6 citations · 10 across the 19 of their papers we have counts for
6 papers · 1 filter
Normalization in Attention Dynamics
Nikita Karagodin, Shu Ge, Yury Polyanskiy +1
We study the effect of normalization schemes on token representations in deep transformers. Modeling their evolution as interacting particles on the sphere, we show that normalizat…
On the Structure of Stationary Solutions to McKean-Vlasov Equations with Applications to Noisy Transformers
Krishnakumar Balasubramanian, Sayan Banerjee, Philippe Rigollet
We study stationary solutions of McKean-Vlasov equations on the circle. Our main contributions stem from observing an exact equivalence between solutions of the stationary McKean-V…
Critical attention scaling in long-context transformers
Shi Chen, Zhengjiang Lin, Yury Polyanskiy +1
As large language models scale to longer contexts, attention layers suffer from a fundamental pathology: attention scores collapse toward uniformity as context length increases…
Synchronization of mean-field models on the circle
Yury Polyanskiy, Philippe Rigollet, Andrew Yao
This paper considers a mean-field model of interacting particles whose state space is the unit circle, a generalization of the classical Kuramoto model. Global synchronization…
Quantitative Clustering in Mean-Field Transformer Models
Shi Chen, Zhengjiang Lin, Yury Polyanskiy +1
The evolution of tokens through deep transformer models can be modeled as an interacting particle system that has been shown to exhibit an asymptotic clustering behavior akin to th…
Residual connections provably mitigate oversmoothing in graph neural networks
Ziang Chen, Zhengjiang Lin, Shi Chen +2
Graph neural networks (GNNs) have achieved remarkable empirical success in processing and representing graph-structured data across various domains. However, a significant challeng…