2 citations · 4 across the 12 of their papers we have counts for
13 papers
Asymptotic Risk of Learning-Rate Schedules for Least-Squares Regression with Discretely Distributed Features
Noé Colomban, Francis Bach, Fabian Schaipp
We study the last iterate of stochastic gradient descent (SGD) on streaming least squares with discretely distributed features (Slock's model), which decouples the dynamics across…
How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size
Fabian Schaipp
We propose a scaling law that takes into account model size and training data while explicitly splitting the latter into training steps and batch size (called three-term law). Fitt…
Sparse regression, classification, and microbial network estimation in QIIME2 with q2-classo and q2-gglasso
Oleg Vlasovets, Fabian Schaipp, Leo Simpson +3
Motivation: Statistical analysis of microbial count data derived from 16S rRNA or metagenomics sequencing poses unique challenges due to the sparse, compositional, and high-dimensi…
Step-Size Stability in Stochastic Optimization: A Theoretical Perspective
Fabian Schaipp, Robert M. Gower, Adrien Taylor
We present a theoretical analysis of stochastic optimization methods in terms of their sensitivity with respect to the step size. We identify a key quantity that, for each method,…
Optimization Benchmark for Diffusion Models on Dynamical Systems
Fabian Schaipp
The training of diffusion models is often absent in the evaluation of new optimization techniques. In this work, we benchmark recent optimization algorithms for training a diffusio…
Analysis of an Idealized Stochastic Polyak Method and its Application to Black-Box Model Distillation
Robert M. Gower, Guillaume Garrigos, Nicolas Loizou +3
We provide a general convergence theorem of an idealized stochastic Polyak step size called SPS. Besides convexity, we only assume a local expected gradient bound, that include…
The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training
Fabian Schaipp, Alexander Hägele, Adrien Taylor +2
We show that learning-rate schedules for large model training behave surprisingly similar to a performance bound from non-smooth convex optimization theory. We provide a bound for…
Tracking the Median of Gradients with a Stochastic Proximal Point Method
Fabian Schaipp, Guillaume Garrigos, Umut Simsekli +1
There are several applications of stochastic optimization where one can benefit from a robust estimate of the gradient. For example, domains such as distributed learning with corru…
Function Value Learning: Adaptive Learning Rates Based on the Polyak Stepsize and Function Splitting in ERM
Guillaume Garrigos, Robert M. Gower, Fabian Schaipp
Here we develop variants of SGD (stochastic gradient descent) with an adaptive step size that make use of the sampled loss values. In particular, we focus on solving a finite sum-o…
MoMo: Momentum Models for Adaptive Learning Rates
Fabian Schaipp, Ruben Ohana, Michael Eickenberg +2
Training a modern machine learning architecture on a new task requires extensive learning-rate tuning, which comes at a high computational cost. Here we develop new Polyak-type ada…
A Stochastic Proximal Polyak Step Size
Fabian Schaipp, Robert M. Gower, Michael Ulbrich
Recently, the stochastic Polyak step size (SPS) has emerged as a competitive adaptive step size scheme for stochastic gradient descent. Here we develop ProxSPS, a proximal variant…
A Semismooth Newton Stochastic Proximal Point Algorithm with Variance Reduction
Andre Milzarek, Fabian Schaipp, Michael Ulbrich
We develop an implementable stochastic proximal point (SPP) method for a class of weakly convex, composite optimization problems. The proposed stochastic proximal point algorithm i…
GGLasso -- a Python package for General Graphical Lasso computation
Fabian Schaipp, Christian L. Müller, Oleg Vlasovets
We introduce GGLasso, a Python package for solving General Graphical Lasso problems. The Graphical Lasso scheme, introduced by (Friedman 2007) (see also (Yuan 2007; Banerjee 2008))…