1 citations · 1 across the 22 of their papers we have counts for
23 papers
Training-Free Transformer Merging via Sequential Local Operator Alignment
Akansh Maurya, Ya-Wei Eileen Lin, Stefanie Jegelka +2
Training-free model merging aims to combine multiple fine-tuned models into a single model without further optimization on labeled data. Yet, in transformers, independently merging…
What's in a Smoothness Constant? Tighter Rates for Local SGD with Bounded Second-order Heterogeneity
Kumar Kshitij Patel, Rustem Islamov, Sebastian U Stich +3
Local SGD, also known as Federated Averaging, is a widely used distributed optimization algorithm. Although Local SGD often outperforms alternatives such as Mini-batch SGD in pract…
The Dual Averaging Power-Prox Method with Application to Heavy-Tail Incremental Gradient
Yuan Gao, Jeremy Rack, Sebastian U. Stich
We study finite-sum composite optimization under two departures from classical stochastic gradient descent theory that are central in practice: incremental gradient access and heav…
Improved Convergence Analysis of Topology Dependence in Decentralized SGD
Yuki Takezawa, Anastasia Koloskova, Sebastian U. Stich
Decentralized SGD is a fundamental algorithm in decentralized learning, although the influence of an underlying network topology on its convergence behavior is not yet fully unders…
Forgetting Has Neighbors: Localized Collateral Forgetting in Machine Unlearning
Polina Dolgova, Sebastian U. Stich
Machine unlearning aims to remove the influence of selected training examples without full retraining. Standard evaluations often summarize unlearning quality with aggregate metric…
Learning When to Adapt
Ali Zindari, Xiaowen Jiang, Rotem Mulayoff +1
Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method, yet its learned correction is static: the same low-rank update is applied to every input. This i…
LoRA vs. Full Fine-Tuning: A Theoretical Perspective
Ali Zindari, Rotem Mulayoff, Sebastian U. Stich
Fine-tuning adapts a pre-trained model to downstream tasks using a small amount of labeled data. Low-Rank Adaptation (LoRA) is an efficient fine-tuning method that reduces memory a…
Efficient Gradient Methods for Distributed Saddle Problems
Ruichen Luo, Anton Rodomanov, Sebastian U. Stich
The distributed setting for Saddle Problems (SPs) has recently emerged as a framework for various modern applications in machine learning and multiagent systems. Despite its releva…
Enhancing LLM Training via Spectral Clipping
Xiaowen Jiang, Andrei Semenov, Sebastian U. Stich
While spectral-based optimizers like Muon operate directly on the spectrum of updates, standard adaptive methods such as AdamW do not account for the spectral structure of weights…
Non-Convex Federated Optimization under Cost-Aware Client Selection
Xiaowen Jiang, Anton Rodomanov, Sebastian U. Stich
Different federated optimization algorithms typically employ distinct client-selection strategies: some methods communicate only with a randomly sampled subset of clients at each r…
Composite Optimization with Error Feedback: the Dual Averaging Approach
Yuan Gao, Anton Rodomanov, Jeremy Rack +1
Communication efficiency is a central challenge in distributed machine learning training, and message compression is a widely used solution. However, standard Error Feedback (EF) m…
FedMuon: Federated Learning with Bias-corrected LMO-based Optimization
Yuki Takezawa, Anastasia Koloskova, Xiaowen Jiang +1
Recently, a new optimization method based on the linear minimization oracle (LMO), called Muon, has been attracting increasing attention since it can train neural networks faster t…
Exploiting Similarity for Computation and Communication-Efficient Decentralized Optimization
Yuki Takezawa, Xiaowen Jiang, Anton Rodomanov +1
Reducing communication complexity is critical for efficient decentralized optimization. The proximal decentralized optimization (PDO) framework is particularly appealing, as method…
Accelerated Distributed Optimization with Compression and Error Feedback
Yuan Gao, Anton Rodomanov, Jeremy Rack +1
Modern machine learning tasks often involve massive datasets and models, necessitating distributed optimization algorithms with reduced communication overhead. Communication compre…
Decoupled SGDA for Games with Intermittent Strategy Communication
Ali Zindari, Parham Yazdkhasti, Anton Rodomanov +2
We focus on reducing communication overhead in multiplayer games, where frequently exchanging strategies between players is not feasible and players have noisy or outdated strategi…
DADA: Dual Averaging with Distance Adaptation
Mohammad Moshtaghifar, Anton Rodomanov, Daniil Vankov +1
We present a novel universal gradient method for solving convex optimization problems. Our algorithm, Dual Averaging with Distance Adaptation (DADA), is based on the classical sche…
Optimizing -Smooth Functions by Gradient Methods
Daniil Vankov, Anton Rodomanov, Angelia Nedich +2
We study gradient methods for optimizing -smooth functions, a class that generalizes Lipschitz-smooth functions and has gained attention for its relevance in machine le…
Stabilized Proximal-Point Methods for Federated Optimization
Xiaowen Jiang, Anton Rodomanov, Sebastian U. Stich
In developing efficient optimization algorithms, it is crucial to account for communication constraints -- a significant challenge in modern Federated Learning. The best-known comm…
Universality of AdaGrad Stepsizes for Stochastic Optimization: Inexact Oracle, Acceleration and Variance Reduction
Anton Rodomanov, Xiaowen Jiang, Sebastian Stich
We present adaptive gradient methods (both basic and accelerated) for solving convex composite optimization problems in which the main part is approximately smooth (a.k.a. …
Towards Faster Decentralized Stochastic Optimization with Communication Compression
Rustem Islamov, Yuan Gao, Sebastian U. Stich
Communication efficiency has garnered significant attention as it is considered the main bottleneck for large-scale decentralized Machine Learning applications in distributed and f…
Federated Optimization with Doubly Regularized Drift Correction
Xiaowen Jiang, Anton Rodomanov, Sebastian U. Stich
Federated learning is a distributed optimization paradigm that allows training machine learning models across decentralized devices while keeping the data localized. The standard m…
Non-convex Stochastic Composite Optimization with Polyak Momentum
Yuan Gao, Anton Rodomanov, Sebastian U. Stich
The stochastic proximal gradient method is a powerful generalization of the widely used stochastic gradient descent (SGD) method and has found numerous applications in Machine Lear…
EControl: Fast Distributed Optimization with Compression and Error Control
Yuan Gao, Rustem Islamov, Sebastian Stich
Modern distributed training relies heavily on communication compression to reduce the communication overhead. In this work, we study algorithms employing a popular class of contrac…