9 citations · 26 across the 18 of their papers we have counts for
5 papers · 1 filter
Perceptrons and localization of attention's mean-field landscape
Antonio Álvarez-López, Borjan Geshkovski, Domènec Ruiz-Balet
The forward pass of a Transformer can be seen as an interacting particle system on the unit sphere: time plays the role of layers, particles that of token embeddings, and the unit…
Dynamic metastability in the self-attention model
Borjan Geshkovski, Hugo Koubbi, Yury Polyanskiy +1
We consider the self-attention model - an interacting particle system on the unit sphere, which serves as a toy model for Transformers, the deep neural network architecture behind…
A mathematical perspective on Transformers
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy +1
Transformers play a central role in the inner workings of large language models. We develop a mathematical framework for analyzing Transformers based on their interpretation as int…
The emergence of clusters in self-attention dynamics
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy +1
Viewing Transformers as interacting particle systems, we describe the geometry of learned representations when the weights are not time dependent. We show that particles, represent…
Sparsity in long-time control of neural ODEs
Carlos Esteve-Yagüe, Borjan Geshkovski
We consider the neural ODE and optimal control perspective of supervised learning, with -control penalties, where rather than only minimizing a final cost (the \emph{empiri…