66 citations · 138 across the 23 of their papers we have counts for
25 papers
Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networks
Alexandre Declèves, Etienne Boursier, Nicolas Flammarion
Adapting pretrained models to downstream tasks with limited data has become a central paradigm in modern deep learning. Yet, despite its widespread practical success, how fine-tuni…
Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networks
Etienne Boursier, Nicolas Flammarion
Fine-tuning pre-trained models on specialized tasks with scarce data is central to modern deep learning. Despite its empirical success, theoretical understanding of fine-tuning rem…
Long-Context Linear System Identification
Oğuz Kaan Yüksel, Mathieu Even, Nicolas Flammarion
This paper addresses the problem of long-context linear system identification, where the state of a dynamical system at time depends linearly on previous states ove…
Simplicity bias and optimization threshold in two-layer ReLU networks
Etienne Boursier, Nicolas Flammarion
Understanding generalization of overparametrized neural networks remains a fundamental challenge in machine learning. Most of the literature mostly studies generalization from an i…
Implicit Bias of Mirror Flow on Separable Data
Scott Pesme, Radu-Alexandru Dragomir, Nicolas Flammarion
We examine the continuous-time counterpart of mirror descent, namely mirror flow, on classification problems which are linearly separable. Such problems are minimised `at infinity'…
Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks
Hristo Papazov, Scott Pesme, Nicolas Flammarion
In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient…
Early alignment in two-layer networks training is a two-edged sword
Etienne Boursier, Nicolas Flammarion
Training neural networks with first order optimisation methods is at the core of the empirical success of deep learning. The scale of initialisation is a crucial factor, as small i…
Saddle-to-Saddle Dynamics in Diagonal Linear Networks
Scott Pesme, Nicolas Flammarion
In this paper we fully describe the trajectory of gradient flow over diagonal linear networks in the limit of vanishing initialisation. We show that the limiting flow successively…
Penalising the biases in norm regularisation enforces sparsity
Etienne Boursier, Nicolas Flammarion
Controlling the parameters' norm often yields good generalisation when training neural networks. Beyond simple intuitions, the relation between regularising parameters' norm and ob…
First-order ANIL provably learns representations despite overparametrization
Oğuz Kaan Yüksel, Etienne Boursier, Nicolas Flammarion
Due to its empirical success in few-shot classification and reinforcement learning, meta-learning has recently received significant interest. Meta-learning methods leverage data fr…
(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability
Mathieu Even, Scott Pesme, Suriya Gunasekar +1
In this paper, we investigate the impact of stochasticity and large stepsizes on the implicit regularisation of gradient descent (GD) and stochastic gradient descent (SGD) over dia…
Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisation
Loucas Pillaud-Vivien, Julien Reygner, Nicolas Flammarion
Understanding the implicit bias of training algorithms is of crucial importance in order to explain the success of overparametrised neural networks. In this paper, we study the rol…
Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs
Etienne Boursier, Loucas Pillaud-Vivien, Nicolas Flammarion
The training of neural networks by gradient descent methods is a cornerstone of the deep learning revolution. Yet, despite some recent progress, a complete theory explaining its su…
Accelerated SGD for Non-Strongly-Convex Least Squares
Aditya Varre, Nicolas Flammarion
We consider stochastic approximation for the least squares regression problem in the non-strongly convex setting. We present the first practical algorithm that achieves the optimal…
Trace norm regularization for multi-task learning with scarce data
Etienne Boursier, Mikhail Konobeev, Nicolas Flammarion
Multi-task learning leverages structural similarities between multiple tasks to learn despite very few samples. Motivated by the recent success of neural networks applied to data-s…
Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity
Scott Pesme, Loucas Pillaud-Vivien, Nicolas Flammarion
Understanding the implicit bias of training algorithms is of crucial importance in order to explain the success of overparametrised neural networks. In this paper, we study the dyn…
A Continuized View on Nesterov Acceleration for Stochastic Gradient Descent and Randomized Gossip
Mathieu Even, Raphaël Berthier, Francis Bach +5
We introduce the continuized Nesterov acceleration, a close variant of Nesterov acceleration whose variables are indexed by a continuous time parameter. The two variables continuou…
A Continuized View on Nesterov Acceleration
Raphaël Berthier, Francis Bach, Nicolas Flammarion +2
We introduce the "continuized" Nesterov acceleration, a close variant of Nesterov acceleration whose variables are indexed by a continuous time parameter. The two variables continu…
Last iterate convergence of SGD for Least-Squares in the Interpolation regime
Aditya Varre, Loucas Pillaud-Vivien, Nicolas Flammarion
Motivated by the recent successes of neural networks that have the ability to fit the data perfectly and generalize well, we study the noiseless model in the fundamental least-squa…
On Convergence-Diagnostic based Step Sizes for Stochastic Gradient Descent
Scott Pesme, Aymeric Dieuleveut, Nicolas Flammarion
Constant step-size Stochastic Gradient Descent exhibits two phases: a transient phase during which iterates make fast progress towards the optimum, followed by a stationary phase d…
Online Robust Regression via SGD on the l1 loss
Scott Pesme, Nicolas Flammarion
We consider the robust linear regression problem in the online setting where we have access to the data in a streaming manner, one data point after the other. More specifically, fo…
Stochastic Composite Least-Squares Regression with convergence rate O(1/n)
Nicolas Flammarion, Francis Bach
We consider the minimization of composite objective functions composed of the expectation of quadratic functions and an arbitrary convex function. We study the stochastic dual aver…
Robust Discriminative Clustering with Sparse Regularizers
Nicolas Flammarion, Balamurugan Palaniappan, Francis Bach
Clustering high-dimensional data often requires some form of dimensionality reduction, where clustered variables are separated from "noise-looking" variables. We cast this problem…
Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regression
Aymeric Dieuleveut, Nicolas Flammarion, Francis Bach
We consider the optimization of a quadratic objective function whose gradients are only accessible through a stochastic oracle that returns the gradient at any given point plus a z…