732 citations · 3.2k across the 65 of their papers we have counts for
46 papers · 1 filter
Optimizers for Diffusion Models: A Controlled Benchmark
Arman Bolatov, Egor Shulgin, David Li +6
Discrete diffusion models now match autoregressive language models on several benchmarks, while the question of how best to train them has received far less attention: the optimize…
On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentials
Rotem Mulayoff, Sebastian U. Stich
The dynamical stability of the iterates during training plays a key role in determining the minima obtained by optimization algorithms. For example, stable solutions of gradient de…
Sequential Subspace Noise Injection Prevents Accuracy Collapse in Certified Unlearning
Polina Dolgova, Sebastian U. Stich
Certified unlearning based on differential privacy offers strong guarantees but remains largely impractical: the noisy fine-tuning approaches proposed so far achieve these guarante…
Scalable Decentralized Learning with Teleportation
Yuki Takezawa, Sebastian U. Stich
Decentralized SGD can run with low communication costs, but its sparse communication characteristics deteriorate the convergence rate, especially when the number of nodes is large.…
The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communication
Kumar Kshitij Patel, Margalit Glasgow, Ali Zindari +5
Local SGD is a popular optimization method in distributed learning, often outperforming other algorithms in practice, including mini-batch SGD. Despite this success, theoretically…
Adaptive SGD with Polyak stepsize and Line-search: Robust Convergence and Variance Reduction
Xiaowen Jiang, Sebastian U. Stich
The recently proposed stochastic Polyak stepsize (SPS) and stochastic line-search (SLS) for SGD have shown remarkable effectiveness when training over-parameterized models. However…
Locally Adaptive Federated Learning
Sohom Mukherjee, Nicolas Loizou, Sebastian U. Stich
Federated learning is a paradigm of distributed machine learning in which multiple clients coordinate with a central server to learn a model, without sharing their own training dat…
Synthetic data shuffling accelerates the convergence of federated learning under data heterogeneity
Bo Li, Yasin Esfandiari, Mikkel N. Schmidt +2
In federated learning, data heterogeneity is a critical challenge. A straightforward solution is to shuffle the clients' data to homogenize the distribution. However, this may viol…
On Convergence of Incremental Gradient for Non-Convex Smooth Functions
Anastasia Koloskova, Nikita Doikov, Sebastian U. Stich +1
In machine learning and neural network optimization, algorithms like incremental gradient, and shuffle SGD are popular due to minimizing the number of cache misses and good practic…
Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees
Anastasia Koloskova, Hadrien Hendrikx, Sebastian U. Stich
Gradient clipping is a popular modification to standard (stochastic) gradient descent, at every iteration limiting the gradient norm to a certain value . It is widely used fo…
On the effectiveness of partial variance reduction in federated learning with heterogeneous data
Bo Li, Mikkel N. Schmidt, Tommy S. Alstrøm +1
Data heterogeneity across clients is a key challenge in federated learning. Prior works address this by either aligning client and server models or using control variates to correc…
Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learning
Anastasia Koloskova, Sebastian U. Stich, Martin Jaggi
We study the asynchronous stochastic gradient descent algorithm for distributed training over workers which have varying computation and communication frequency over time. In t…
Special Properties of Gradient Descent with Large Learning Rates
Amirkeivan Mohtashami, Martin Jaggi, Sebastian Stich
When training neural networks, it has been widely observed that a large step size is essential in stochastic gradient descent (SGD) for obtaining superior models. However, the effe…
Data-heterogeneity-aware Mixing for Decentralized Learning
Yatin Dandi, Anastasia Koloskova, Martin Jaggi +1
Decentralized learning provides an effective framework to train machine learning models with data distributed over arbitrary communication graphs. However, most existing approaches…
Tackling benign nonconvexity with smoothing and stochastic gradients
Harsh Vardhan, Sebastian U. Stich
Non-convex optimization problems are ubiquitous in machine learning, especially in Deep Learning. While such complex problems can often be successfully optimized in practice by usi…
Characterizing & Finding Good Data Orderings for Fast Convergence of Sequential Gradient Methods
Amirkeivan Mohtashami, Sebastian Stich, Martin Jaggi
While SGD, which samples from the data with replacement is widely studied in theory, a variant called Random Reshuffling (RR) is more common in practice. RR iterates through random…
ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!
Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich +1
We introduce ProxSkip -- a surprisingly simple and provably efficient method for minimizing the sum of a smooth () and an expensive nonsmooth proximable () function. The cano…
The Peril of Popular Deep Learning Uncertainty Estimation Methods
Yehao Liu, Matteo Pagliardini, Tatjana Chavdarova +1
Uncertainty estimation (UE) techniques -- such as the Gaussian process (GP), Bayesian neural networks (BNN), Monte Carlo dropout (MCDropout) -- aim to improve the interpretability…
Linear Speedup in Personalized Collaborative Learning
El Mahdi Chayti, Sai Praneeth Karimireddy, Sebastian U. Stich +2
Collaborative training can improve the accuracy of a model for a user by trading off the model's bias (introduced by using data from other users who are potentially different) agai…
ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training
Hui-Po Wang, Sebastian U. Stich, Yang He +1
Federated learning is a powerful distributed learning scheme that allows numerous edge devices to collaboratively train a model without sharing their data. However, training is res…
RelaySum for Decentralized Deep Learning on Heterogeneous Data
Thijs Vogels, Lie He, Anastasia Koloskova +4
In decentralized machine learning, workers compute model updates on their local data. Because the workers only communicate with few neighbors without central coordination, these up…
On Second-order Optimization Methods for Federated Learning
Sebastian Bischoff, Stephan Günnemann, Martin Jaggi +1
We consider federated learning (FL), where the training data is distributed across a large number of clients. The standard optimization method in this setting is Federated Averagin…
A Field Guide to Federated Optimization
Jianyu Wang, Zachary Charles, Zheng Xu +50
Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy prote…
Masked Training of Neural Networks with Partial Gradients
Amirkeivan Mohtashami, Martin Jaggi, Sebastian U. Stich
State-of-the-art training algorithms for deep learning models are based on stochastic gradient descent (SGD). Recently, many variations have been explored: perturbing parameters fo…