6 citations · 10 across the 19 of their papers we have counts for
20 papers
LEGO: Synergizing Expert GraphRAG and Expert Chain-of-Thought for Legal Reasoning
Qingjing Chen, Junkai Zhang, Shaochun Wang +7
Large language models are increasingly applied to high-risk domains such as law, yet complex legal reasoning remains limited by two structural challenges. First, existing RAG and G…
Quantitative Diffusive Limits for Singular Nonlocal Transport
Andrea Agazzi, Giuseppe Bruno, Federico Pasqualotto +1
We study the nonlocal continuity equation \[ \partial_tμ_b =\operatorname{div}\!\left( μ_b\nabla\log\bigl((I-b^2Δ)^{-1}μ_b\bigr) \right) \] on a closed connected Riemannian manifol…
Measure-to-measure Regression with Transformers
Matthew Vandergrift, Martha White, Yury Polyanskiy +2
Many learning problems require predicting how populations evolve under an unknown transformation. A natural representation for such populations is a probability measure, with point…
Propagation of Chaos in Contextual Flow Maps
Shi Chen, Zhengjiang Lin, Kaizhao Liu +1
We develop a quantitative statistical theory of transformers in the large-context regime by adopting the abstraction of contextual flow maps (CFMs): dynamical systems that evolve a…
Scaling Limits of Long-Context Transformers
Giuseppe Bruno, Shi Chen, Zhengjiang Lin +2
We study the long-context limit of softmax self-attention with a fixed query and a random context of i.i.d. keys on the sphere, viewing the inverse temperature as the sca…
Homogenized Transformers
Hugo Koubbi, Borjan Geshkovski, Philippe Rigollet
We study a random model of deep multi-head self-attention in which the weights are resampled independently across layers and heads, as at initialization of training. Viewing depth…
YuriiFormer: A Suite of Nesterov-Accelerated Transformers
Aleksandr Zimin, Yury Polyanskiy, Philippe Rigollet
We propose a variational framework that interprets transformer layers as iterations of an optimization algorithm acting on token embeddings. In this view, self-attention implements…
The Mean-Field Dynamics of Transformers
Philippe Rigollet
We develop a mathematical framework that interprets Transformer attention as an interacting particle system and studies its continuum (mean-field) limits. By idealizing attention o…
Normalization in Attention Dynamics
Nikita Karagodin, Shu Ge, Yury Polyanskiy +1
We study the effect of normalization schemes on token representations in deep transformers. Modeling their evolution as interacting particles on the sphere, we show that normalizat…
On the Structure of Stationary Solutions to McKean-Vlasov Equations with Applications to Noisy Transformers
Krishnakumar Balasubramanian, Sayan Banerjee, Philippe Rigollet
We study stationary solutions of McKean-Vlasov equations on the circle. Our main contributions stem from observing an exact equivalence between solutions of the stationary McKean-V…
Critical attention scaling in long-context transformers
Shi Chen, Zhengjiang Lin, Yury Polyanskiy +1
As large language models scale to longer contexts, attention layers suffer from a fundamental pathology: attention scores collapse toward uniformity as context length increases…
Synchronization of mean-field models on the circle
Yury Polyanskiy, Philippe Rigollet, Andrew Yao
This paper considers a mean-field model of interacting particles whose state space is the unit circle, a generalization of the classical Kuramoto model. Global synchronization…
Quantitative Clustering in Mean-Field Transformer Models
Shi Chen, Zhengjiang Lin, Yury Polyanskiy +1
The evolution of tokens through deep transformer models can be modeled as an interacting particle system that has been shown to exhibit an asymptotic clustering behavior akin to th…
Residual connections provably mitigate oversmoothing in graph neural networks
Ziang Chen, Zhengjiang Lin, Shi Chen +2
Graph neural networks (GNNs) have achieved remarkable empirical success in processing and representing graph-structured data across various domains. However, a significant challeng…
On the number of modes of Gaussian kernel density estimators
Borjan Geshkovski, Philippe Rigollet, Yihang Sun
We consider the Gaussian kernel density estimator with bandwidth of iid Gaussian samples. Using the Kac-Rice formula and an Edgeworth expansion, we prove that th…
Clustering in Causal Attention Masking
Nikita Karagodin, Yury Polyanskiy, Philippe Rigollet
This work presents a modification of the self-attention dynamics proposed by Geshkovski et al. (arXiv:2312.10794) to better reflect the practically relevant, causally masked attent…
Measure-to-measure interpolation using Transformers
Borjan Geshkovski, Philippe Rigollet, Domènec Ruiz-Balet
Transformers are deep neural network architectures that underpin the recent successes of large language models. Unlike more classical architectures that can be viewed as point-to-p…
Dynamic metastability in the self-attention model
Borjan Geshkovski, Hugo Koubbi, Yury Polyanskiy +1
We consider the self-attention model - an interacting particle system on the unit sphere, which serves as a toy model for Transformers, the deep neural network architecture behind…
A mathematical perspective on Transformers
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy +1
Transformers play a central role in the inner workings of large language models. We develop a mathematical framework for analyzing Transformers based on their interpretation as int…
Covariance alignment: from maximum likelihood estimation to Gromov-Wasserstein
Yanjun Han, Philippe Rigollet, George Stepaniants
Feature alignment methods are used in many scientific disciplines for data pooling, annotation, and comparison. As an instance of a permutation learning problem, feature alignment…