13 papers
MARS: Malware Analysis with Rule-Based Scoring of LLM Claims
Hyeongjun Choi
Large language models can triage malware through direct verdicts or behavioral claims scored by an external policy. We present MARS, a malware triage framework, and compare direct…
Universality and Generalization of Causal Transformers Across Context Lengths
Takashi Furuya, Maarten V. de Hoop, Gabriel Peyré
Long contexts are central to modern transformer systems, but most expressivity results choose a different network for each fixed sequence length. We study whether one masked transf…
Sample complexity of unbalanced entropic OT
Francisco Andrade, Gabriel Peyré, Clarice Poon
Optimal transport (OT) has become a central language for comparing probability measures, but exact balanced OT is often both too rigid for data with missing, created, or destroyed…
Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence
Valérie Castin, Kimia Nadjahi, Pierre Ablin +1
Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models. Notably, LoRA is inherently overparameterized: multiple pairs of low-rank factor…
Training Infinitely Deep and Wide Transformers
Raphaël Barboni, Maarten V. de Hoop, Takashi Furuya +1
Transformers have become the dominant architecture in modern machine learning, yet the theoretical understanding of their training dynamics remains limited. This paper develops a r…
Geometry-Aware Discretization Error of Diffusion Models
Samuel Hurault, Thomas Moreau, Gabriel Peyré
Practical diffusion sampling requires simulating a reverse-time ODE or SDE with a limited number of denoising steps, making the choice of sampling parameters crucial for minimizing…
Muon Dynamics as a Spectral Wasserstein Flow
Gabriel Peyré
Gradient normalization can stabilize deep-learning optimization, and spectral normalizations are especially natural for matrix-shaped parameter blocks; Muon is the motivating examp…
On the global convergence of gradient flow for wide shallow models beyond homogeneous nonlinearities
Romain Petit, Clarice Poon, Gabriel Peyré
A surprising phenomenon in the training of neural networks is the ability of gradient descent to find global minimizers of the training loss despite its non-convexity. Following ea…
Second-order geometry of optimal transport: cost identifiability and stability
Gabriel Peyré, Clarice Poon, Oscar Tron
We study the optimal transport value as a functional of the ground cost. Although this functional is an infimum of linear functionals and is therefore polyhedral for finitely suppo…
Robust Sublinear Convergence Rates for Iterative Bregman Projections
Gabriel Peyré
Entropic regularization provides a simple way to approximate linear programs whose constraints split into two or more tractable blocks. The resulting objectives are amenable to cyc…
Token Sample Complexity of Attention
Léa Bohbot, Cyril Letrouit, Gabriel Peyré +1
As context windows in large language models continue to expand, it is essential to characterize how attention behaves at extreme sequence lengths. We introduce token sample complex…
Optimal and Diffusion Transports in Machine Learning
Gabriel Peyré
Several problems in machine learning are naturally expressed as the design and analysis of time-evolving probability distributions. This includes sampling via diffusion methods, op…
Intrinsic training dynamics of deep neural networks
Sibylle Marcotte, Gabriel Peyré, Rémi Gribonval
A fundamental challenge in the theory of deep learning is to understand whether gradient-based training can promote parameters belonging to certain lower-dimensional structures (e.…