9 citations · 26 across the 18 of their papers we have counts for
18 papers
A Lloyd-stabilized Voronoi particle method
Bruno Després, Borjan Geshkovski
We study the stability and consistency of Lloyd's algorithm used in combination with the Lagrangian transport of a density on a Voronoï tessellation. We show that relaxation rates…
Kinetic theory for Transformers and the lost-in-the-middle phenomenon
Mitia Duerinckx, Borjan Geshkovski, Stefano Rossi
We study causal self-attention dynamics -- a toy model for decoder Transformers -- which we interpret as a non-exchangeable interacting particle system. Adapting cumulant expansion…
Homogenized Transformers
Hugo Koubbi, Borjan Geshkovski, Philippe Rigollet
We study a random model of deep multi-head self-attention in which the weights are resampled independently across layers and heads, as at initialization of training. Viewing depth…
Constructive conditional normalizing flows
Borjan Geshkovski, Domènec Ruiz-Balet
Motivated by applications in conditional sampling, given a probability measure and a diffeomorphism , we consider the problem of simultaneously approximating and the pus…
Perceptrons and localization of attention's mean-field landscape
Antonio Álvarez-López, Borjan Geshkovski, Domènec Ruiz-Balet
The forward pass of a Transformer can be seen as an interacting particle system on the unit sphere: time plays the role of layers, particles that of token embeddings, and the unit…
Attention's forward pass and Frank-Wolfe
Albert Alcalde, Borjan Geshkovski, Domènec Ruiz-Balet
We study the hardmax limit of self-attention dynamics for token embeddings obtained in the zero-temperature () regime, and relate it to the finite- setting. In this…
Constructive approximate transport maps with normalizing flows
Antonio Álvarez-López, Borjan Geshkovski, Domènec Ruiz-Balet
We study an approximate controllability problem for the continuity equation and its application to constructing transport maps with normalizing flows. Specifically, we construct ti…
On the number of modes of Gaussian kernel density estimators
Borjan Geshkovski, Philippe Rigollet, Yihang Sun
We consider the Gaussian kernel density estimator with bandwidth of iid Gaussian samples. Using the Kac-Rice formula and an Edgeworth expansion, we prove that th…
Measure-to-measure interpolation using Transformers
Borjan Geshkovski, Philippe Rigollet, Domènec Ruiz-Balet
Transformers are deep neural network architectures that underpin the recent successes of large language models. Unlike more classical architectures that can be viewed as point-to-p…
Dynamic metastability in the self-attention model
Borjan Geshkovski, Hugo Koubbi, Yury Polyanskiy +1
We consider the self-attention model - an interacting particle system on the unit sphere, which serves as a toy model for Transformers, the deep neural network architecture behind…
A mathematical perspective on Transformers
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy +1
Transformers play a central role in the inner workings of large language models. We develop a mathematical framework for analyzing Transformers based on their interpretation as int…
The emergence of clusters in self-attention dynamics
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy +1
Viewing Transformers as interacting particle systems, we describe the geometry of learned representations when the weights are not time dependent. We show that particles, represent…
Control of the Stefan problem in a periodic box
Borjan Geshkovski, Debayan Maity
In this paper we consider the one-phase Stefan problem with surface tension, set in a two-dimensional strip-like geometry, with periodic boundary conditions respect to the horizont…
Turnpike in optimal control of PDEs, ResNets, and beyond
Borjan Geshkovski, Enrique Zuazua
The \emph{turnpike property} in contemporary macroeconomics asserts that if an economic planner seeks to move an economy from one level of capital to another, then the most efficie…
Optimal actuator design via Brunovsky's normal form
Borjan Geshkovski, Enrique Zuazua
In this paper, by using the Brunovsky normal form, we provide a reformulation of the problem consisting in finding the actuator design which minimizes the controllability cost for…
Sparsity in long-time control of neural ODEs
Carlos Esteve-Yagüe, Borjan Geshkovski
We consider the neural ODE and optimal control perspective of supervised learning, with -control penalties, where rather than only minimizing a final cost (the \emph{empiri…
Turnpike in Lipschitz-nonlinear optimal control
Carlos Esteve-Yagüe, Borjan Geshkovski, Dario Pighin +1
We present a new proof of the turnpike property for nonlinear optimal control problems, when the running target is a steady control-state pair of the underlying system. Our strateg…
Large-time asymptotics in deep learning
Carlos Esteve, Borjan Geshkovski, Dario Pighin +1
We consider the neural ODE perspective of supervised learning and study the impact of the final time (which may indicate the depth of a corresponding ResNet) in training. For t…