9 citations · 9 across the 5 of their papers we have counts for
5 papers
High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Model
O. Duranthon, F. Boncoraglio, L. Zdeborová
We develop a high-dimensional statistical theory of low-rank adaptation (LoRA) in attention models, capturing the interplay between pre-training and fine-tuning. We introduce a sol…
Statistical Advantage of Softmax Attention: Insights from Single-Location Regression
O. Duranthon, P. Marion, C. Boyer +2
Large language models rely on attention mechanisms with a softmax activation. Yet the dominance of softmax over alternatives (e.g., component-wise or linear) remains poorly underst…
Statistical physics analysis of graph neural networks: Approaching optimality in the contextual stochastic block model
O. Duranthon, L. Zdeborová
Graph neural networks (GNNs) are designed to process data associated with graphs. They are finding an increasing range of applications; however, as with other modern machine learni…
Coarse-grained quantum cellular automata
O. Duranthon, Giuseppe Di Molfetta
One can think of some physical evolutions as being the emergent-effective result of a microscopic discrete model. Inspired by classical coarse-graining procedures, we provide a sim…
Maximal Relevance and Optimal Learning Machines
O Duranthon, M Marsili, R Xie
We show that the mutual information between the representation of a learning machine and the hidden features that it extracts from data is bounded from below by the relevance, whic…