5 papers
Asymptotic Risk of Learning-Rate Schedules for Least-Squares Regression with Discretely Distributed Features
Noé Colomban, Francis Bach, Fabian Schaipp
We study the last iterate of stochastic gradient descent (SGD) on streaming least squares with discretely distributed features (Slock's model), which decouples the dynamics across…
The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training
Fabian Schaipp, Alexander Hägele, Adrien Taylor +2
We show that learning-rate schedules for large model training behave surprisingly similar to a performance bound from non-smooth convex optimization theory. We provide a bound for…
Constructive approaches to concentration inequalities with independent random variables
Celine Moucer, Adrien Taylor, Francis Bach
Concentration inequalities, a major tool in probability theory, quantify how much a random variable deviates from a certain quantity. This paper proposes a systematic convex optimi…
Geometry-dependent matching pursuit: a transition phase for convergence on linear regression and LASSO
Céline Moucer, Adrien Taylor, Francis Bach
Greedy first-order methods, such as coordinate descent with Gauss-Southwell rule or matching pursuit, have become popular in optimization due to their natural tendency to propose s…
A note on approximate accelerated forward-backward methods with absolute and relative errors, and possibly strongly convex objectives
Mathieu Barré, Adrien Taylor, Francis Bach
In this short note, we provide a simple version of an accelerated forward-backward method (a.k.a. Nesterov's accelerated proximal gradient method) possibly relying on approximate p…