9 citations · 26 across the 18 of their papers we have counts for
5 papers · 1 filter
A Lloyd-stabilized Voronoi particle method
Bruno Després, Borjan Geshkovski
We study the stability and consistency of Lloyd's algorithm used in combination with the Lagrangian transport of a density on a Voronoï tessellation. We show that relaxation rates…
Kinetic theory for Transformers and the lost-in-the-middle phenomenon
Mitia Duerinckx, Borjan Geshkovski, Stefano Rossi
We study causal self-attention dynamics -- a toy model for decoder Transformers -- which we interpret as a non-exchangeable interacting particle system. Adapting cumulant expansion…
Homogenized Transformers
Hugo Koubbi, Borjan Geshkovski, Philippe Rigollet
We study a random model of deep multi-head self-attention in which the weights are resampled independently across layers and heads, as at initialization of training. Viewing depth…
Constructive conditional normalizing flows
Borjan Geshkovski, Domènec Ruiz-Balet
Motivated by applications in conditional sampling, given a probability measure and a diffeomorphism , we consider the problem of simultaneously approximating and the pus…
Perceptrons and localization of attention's mean-field landscape
Antonio Álvarez-López, Borjan Geshkovski, Domènec Ruiz-Balet
The forward pass of a Transformer can be seen as an interacting particle system on the unit sphere: time plays the role of layers, particles that of token embeddings, and the unit…