66 citations · 138 across the 23 of their papers we have counts for
10 papers · 1 filter
Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networks
Alexandre Declèves, Etienne Boursier, Nicolas Flammarion
Adapting pretrained models to downstream tasks with limited data has become a central paradigm in modern deep learning. Yet, despite its widespread practical success, how fine-tuni…
Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networks
Etienne Boursier, Nicolas Flammarion
Fine-tuning pre-trained models on specialized tasks with scarce data is central to modern deep learning. Despite its empirical success, theoretical understanding of fine-tuning rem…
Long-Context Linear System Identification
Oğuz Kaan Yüksel, Mathieu Even, Nicolas Flammarion
This paper addresses the problem of long-context linear system identification, where the state of a dynamical system at time depends linearly on previous states ove…
Implicit Bias of Mirror Flow on Separable Data
Scott Pesme, Radu-Alexandru Dragomir, Nicolas Flammarion
We examine the continuous-time counterpart of mirror descent, namely mirror flow, on classification problems which are linearly separable. Such problems are minimised `at infinity'…
Penalising the biases in norm regularisation enforces sparsity
Etienne Boursier, Nicolas Flammarion
Controlling the parameters' norm often yields good generalisation when training neural networks. Beyond simple intuitions, the relation between regularising parameters' norm and ob…
Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisation
Loucas Pillaud-Vivien, Julien Reygner, Nicolas Flammarion
Understanding the implicit bias of training algorithms is of crucial importance in order to explain the success of overparametrised neural networks. In this paper, we study the rol…
Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs
Etienne Boursier, Loucas Pillaud-Vivien, Nicolas Flammarion
The training of neural networks by gradient descent methods is a cornerstone of the deep learning revolution. Yet, despite some recent progress, a complete theory explaining its su…
Trace norm regularization for multi-task learning with scarce data
Etienne Boursier, Mikhail Konobeev, Nicolas Flammarion
Multi-task learning leverages structural similarities between multiple tasks to learn despite very few samples. Motivated by the recent success of neural networks applied to data-s…
Robust Discriminative Clustering with Sparse Regularizers
Nicolas Flammarion, Balamurugan Palaniappan, Francis Bach
Clustering high-dimensional data often requires some form of dimensionality reduction, where clustered variables are separated from "noise-looking" variables. We cast this problem…
From Averaging to Acceleration, There is Only a Step-size
Nicolas Flammarion, Francis Bach
We show that accelerated gradient descent, averaged gradient descent and the heavy-ball method for non-strongly-convex problems may be reformulated as constant parameter second-ord…