7 papers
The Well-Tempered Classifier: Some Elementary Properties of Temperature Scaling
Pierre-Alexandre Mattei, Bruno Loureiro
Temperature scaling is a simple method that allows to control the uncertainty of probabilistic models. It is mostly used in two contexts: improving the calibration of classifiers a…
When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index models
Gibbs Nwemadji, Bruno Loureiro, Jean Barbier
Pre-training on a source task is usually expected to facilitate fine-tuning on similar downstream problems. In this work, we mathematically show that this naive intuition is not al…
A Random Matrix Theory of Masked Self-Supervised Regression
Arie Wortsman Zurich, Federica Gerace, Bruno Loureiro +1
In the era of transformer models, masked self-supervised learning (SSL) has become a foundational training paradigm. A defining feature of masked SSL is that training aggregates pr…
Optical kernel machine with programmable nonlinearity
SeungYun Han, Fei Xia, Sylvain Gigan +2
Optical kernel machines offer high throughput and low latency. A nonlinear optical kernel can handle complex nonlinear data, but power consumption is typically high with the conven…
Dynamical mean-field analysis of adaptive Langevin diffusions: Replica-symmetric fixed point and empirical Bayes
Zhou Fan, Justin Ko, Bruno Loureiro +2
In many applications of statistical estimation via sampling, one may wish to sample from a high-dimensional target distribution that is adaptively evolving to the samples already s…
Dynamical mean-field analysis of adaptive Langevin diffusions: Propagation-of-chaos and convergence of the linear response
Zhou Fan, Justin Ko, Bruno Loureiro +2
Motivated by an application to empirical Bayes learning in high-dimensional regression, we study a class of Langevin diffusions in a system with random disorder, where the drift co…