6 citations · 9 across the 14 of their papers we have counts for
16 papers
YuriiFormer: A Suite of Nesterov-Accelerated Transformers
Aleksandr Zimin, Yury Polyanskiy, Philippe Rigollet
We propose a variational framework that interprets transformer layers as iterations of an optimization algorithm acting on token embeddings. In this view, self-attention implements…
Normalization in Attention Dynamics
Nikita Karagodin, Shu Ge, Yury Polyanskiy +1
We study the effect of normalization schemes on token representations in deep transformers. Modeling their evolution as interacting particles on the sphere, we show that normalizat…
Critical attention scaling in long-context transformers
Shi Chen, Zhengjiang Lin, Yury Polyanskiy +1
As large language models scale to longer contexts, attention layers suffer from a fundamental pathology: attention scores collapse toward uniformity as context length increases…
Global Minimizers of Sigmoid Contrastive Loss
Kiril Bangachev, Guy Bresler, Iliyas Noman +1
The meta-task of obtaining and aligning representations through contrastive pretraining is steadily gaining importance since its introduction in CLIP and ALIGN. In this paper we th…
Synchronization of mean-field models on the circle
Yury Polyanskiy, Philippe Rigollet, Andrew Yao
This paper considers a mean-field model of interacting particles whose state space is the unit circle, a generalization of the classical Kuramoto model. Global synchronization…
Quantitative Clustering in Mean-Field Transformer Models
Shi Chen, Zhengjiang Lin, Yury Polyanskiy +1
The evolution of tokens through deep transformer models can be modeled as an interacting particle system that has been shown to exhibit an asymptotic clustering behavior akin to th…
Partial and Exact Recovery of a Random Hypergraph from its Graph Projection
Guy Bresler, Chenghao Guo, Yury Polyanskiy +1
Consider a -uniform random hypergraph on vertices in which hyperedges are included iid so that the average degree is . The projection of a hypergraph is a graph on the…
Thresholds for Reconstruction of Random Hypergraphs From Graph Projections
Guy Bresler, Chenghao Guo, Yury Polyanskiy
The graph projection of a hypergraph is a simple graph with the same vertex set and with an edge between each pair of vertices that appear in a hyperedge. We consider the problem o…
Residual connections provably mitigate oversmoothing in graph neural networks
Ziang Chen, Zhengjiang Lin, Shi Chen +2
Graph neural networks (GNNs) have achieved remarkable empirical success in processing and representing graph-structured data across various domains. However, a significant challeng…
Clustering in Causal Attention Masking
Nikita Karagodin, Yury Polyanskiy, Philippe Rigollet
This work presents a modification of the self-attention dynamics proposed by Geshkovski et al. (arXiv:2312.10794) to better reflect the practically relevant, causally masked attent…
Dynamic metastability in the self-attention model
Borjan Geshkovski, Hugo Koubbi, Yury Polyanskiy +1
We consider the self-attention model - an interacting particle system on the unit sphere, which serves as a toy model for Transformers, the deep neural network architecture behind…
Entropy Contractions in Markov Chains: Half-Step, Full-Step and Continuous-Time
Pietro Caputo, Zongchen Chen, Yuzhou Gu +1
This paper considers the speed of convergence (mixing) of a finite Markov kernel with respect to the Kullback-Leibler divergence (entropy). Given a Markov kernel one defines ei…
Unsourced Multiple Access: A Coding Paradigm for Massive Random Access
Gianluigi Liva, Yury Polyanskiy
This paper is a tutorial introduction to the field of unsourced multiple access (UMAC) protocols. We first provide a historical survey of the evolution of random access protocols,…
Evolution of the 5G New Radio Two-Step Random Access towards 6G Unsourced MAC
Patrick Agostini, Jean-Francois Chamberland, Federico Clazzer +7
This report summarizes some considerations on possible evolutions of grant-free random access in the next generation of the 3GPP wireless cellular standard. The analysis is carried…
A mathematical perspective on Transformers
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy +1
Transformers play a central role in the inner workings of large language models. We develop a mathematical framework for analyzing Transformers based on their interpretation as int…
Attracting Random Walks
Julia Gaudio, Yury Polyanskiy
This paper introduces the Attracting Random Walks model, which describes the dynamics of a system of particles on a graph with vertices. At each step, a single particle moves t…