6 papers
What is the long-run distribution of stochastic gradient descent? A large deviations analysis
Waïss Azizian, Franck Iutzeler, Jérôme Malick +1
In this paper, we examine the long-run distribution of stochastic gradient descent (SGD) in general, non-convex problems. Specifically, we seek to understand which regions of the p…
: a library for Wasserstein distributionally robust machine learning
Florian Vincent, Waïss Azizian, Franck Iutzeler +1
We present skwdro, a Python library for training robust machine learning models. The library is based on distributionally robust optimization using Wasserstein distances, popular i…
The global convergence time of stochastic gradient descent in non-convex landscapes: Sharp estimates via large deviations
Waïss Azizian, Franck Iutzeler, Jérôme Malick +1
In this paper, we examine the time it takes for stochastic gradient descent (SGD) to reach the global minimum of a general, non-convex loss function. We approach this question thro…
The rate of convergence of Bregman proximal methods: Local geometry vs. regularity vs. sharpness
Waïss Azizian, Franck Iutzeler, Jérôme Malick +1
We examine the last-iterate convergence rate of Bregman proximal methods - from mirror descent to mirror-prox and its optimistic variants - as a function of the local geometry indu…
Derivatives of Stochastic Gradient Descent in parametric optimization
Franck Iutzeler, Edouard Pauwels, Samuel Vaiter
We consider stochastic optimization problems where the objective depends on some parameter, as commonly found in hyperparameter optimization for instance. We investigate the behavi…
Delay-tolerant distributed Bregman proximal algorithms
S. Chraibi, F. Iutzeler, J. Malick +1
Many problems in machine learning write as the minimization of a sum of individual loss functions over the training examples. These functions are usually differentiable but, in som…