11 papers
Separators Make Carry Propagation Learnable:The Geometry of Latent Carry in a Multiplication Transformer
Sama Satariyan, Raphael Cousin, G{é}rard Biau
Transformers asked to multiply multi-digit numbers in a single forward pass often fail, and interpretability studies of pretrained language models find arithmetic solved by input-r…
Distilling LLM Feedback for Lean Theorem Proving
Gaetan Narozniak, Gérard Biau, Rémi Munos +2
Post-training for reasoning models typically combines supervised fine-tuning with reinforcement learning from verifiable rewards, most commonly with GRPO. However, this algorithm s…
A Note on k-NN Gating in RAG
Gérard Biau, Claire Boyer
We propose a statistical proxy framework for retrieval-augmented generation (RAG) that formalizes how language models balance internal predictions with retrieved evidence. We deriv…
Optimal Stopping in Latent Diffusion Models
Yu-Han Wu, Quentin Berthet, Gérard Biau +3
We identify and analyze a surprising phenomenon of Latent Diffusion Models (LDMs) where the final steps of the diffusion can degrade sample quality. In contrast to conventional arg…
Fast kernel methods: Sobolev, physics-informed, and additive models
Nathan Doumèche, Francis Bach, Gérard Biau +1
Kernel methods are powerful tools in statistical learning, but their cubic complexity in the sample size n limits their use on large-scale datasets. In this work, we introduce a sc…
Forecasting time series with constraints
Nathan Doumèche, Francis Bach, Éloi Bedek +3
Time series forecasting presents unique challenges that limit the effectiveness of traditional machine learning algorithms. To address these limitations, various approaches have in…
Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorization
Yu-Han Wu, Pierre Marion, Gérard Biau +1
Denoising score matching plays a pivotal role in the performance of diffusion-based generative models. However, the empirical optimal score--the exact solution to the denoising sco…
Attention layers provably solve single-location regression
Pierre Marion, Raphaël Berthier, Gérard Biau +1
Attention-based models, such as Transformer, excel across various tasks but lack a comprehensive theoretical understanding, especially regarding token-wise sparsity and internal li…
Physics-informed kernel learning
Nathan Doumèche, Francis Bach, Gérard Biau +1
Physics-informed machine learning typically integrates physical priors into the learning process by minimizing a loss function that includes both a data-driven term and a partial d…
Physics-informed machine learning as a kernel method
Nathan Doumèche, Francis Bach, Gérard Biau +1
Physics-informed machine learning combines the expressiveness of data-based approaches with the interpretability of physical models. In this context, we consider a general regressi…
Implicit regularization of deep residual networks towards neural ODEs
Pierre Marion, Yu-Han Wu, Michael E. Sander +1
Residual neural networks are state-of-the-art deep learning models. Their continuous-depth analog, neural ordinary differential equations (ODEs), are also widely used. Despite thei…