1 citations · 1 across the 22 of their papers we have counts for
14 papers · 1 filter
Training-Free Transformer Merging via Sequential Local Operator Alignment
Akansh Maurya, Ya-Wei Eileen Lin, Stefanie Jegelka +2
Training-free model merging aims to combine multiple fine-tuned models into a single model without further optimization on labeled data. Yet, in transformers, independently merging…
What's in a Smoothness Constant? Tighter Rates for Local SGD with Bounded Second-order Heterogeneity
Kumar Kshitij Patel, Rustem Islamov, Sebastian U Stich +3
Local SGD, also known as Federated Averaging, is a widely used distributed optimization algorithm. Although Local SGD often outperforms alternatives such as Mini-batch SGD in pract…
Improved Convergence Analysis of Topology Dependence in Decentralized SGD
Yuki Takezawa, Anastasia Koloskova, Sebastian U. Stich
Decentralized SGD is a fundamental algorithm in decentralized learning, although the influence of an underlying network topology on its convergence behavior is not yet fully unders…
Forgetting Has Neighbors: Localized Collateral Forgetting in Machine Unlearning
Polina Dolgova, Sebastian U. Stich
Machine unlearning aims to remove the influence of selected training examples without full retraining. Standard evaluations often summarize unlearning quality with aggregate metric…
Learning When to Adapt
Ali Zindari, Xiaowen Jiang, Rotem Mulayoff +1
Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method, yet its learned correction is static: the same low-rank update is applied to every input. This i…
LoRA vs. Full Fine-Tuning: A Theoretical Perspective
Ali Zindari, Rotem Mulayoff, Sebastian U. Stich
Fine-tuning adapts a pre-trained model to downstream tasks using a small amount of labeled data. Low-Rank Adaptation (LoRA) is an efficient fine-tuning method that reduces memory a…
Enhancing LLM Training via Spectral Clipping
Xiaowen Jiang, Andrei Semenov, Sebastian U. Stich
While spectral-based optimizers like Muon operate directly on the spectrum of updates, standard adaptive methods such as AdamW do not account for the spectral structure of weights…
Non-Convex Federated Optimization under Cost-Aware Client Selection
Xiaowen Jiang, Anton Rodomanov, Sebastian U. Stich
Different federated optimization algorithms typically employ distinct client-selection strategies: some methods communicate only with a randomly sampled subset of clients at each r…
FedMuon: Federated Learning with Bias-corrected LMO-based Optimization
Yuki Takezawa, Anastasia Koloskova, Xiaowen Jiang +1
Recently, a new optimization method based on the linear minimization oracle (LMO), called Muon, has been attracting increasing attention since it can train neural networks faster t…
Exploiting Similarity for Computation and Communication-Efficient Decentralized Optimization
Yuki Takezawa, Xiaowen Jiang, Anton Rodomanov +1
Reducing communication complexity is critical for efficient decentralized optimization. The proximal decentralized optimization (PDO) framework is particularly appealing, as method…
Decoupled SGDA for Games with Intermittent Strategy Communication
Ali Zindari, Parham Yazdkhasti, Anton Rodomanov +2
We focus on reducing communication overhead in multiplayer games, where frequently exchanging strategies between players is not feasible and players have noisy or outdated strategi…
Stabilized Proximal-Point Methods for Federated Optimization
Xiaowen Jiang, Anton Rodomanov, Sebastian U. Stich
In developing efficient optimization algorithms, it is crucial to account for communication constraints -- a significant challenge in modern Federated Learning. The best-known comm…
Towards Faster Decentralized Stochastic Optimization with Communication Compression
Rustem Islamov, Yuan Gao, Sebastian U. Stich
Communication efficiency has garnered significant attention as it is considered the main bottleneck for large-scale decentralized Machine Learning applications in distributed and f…
Federated Optimization with Doubly Regularized Drift Correction
Xiaowen Jiang, Anton Rodomanov, Sebastian U. Stich
Federated learning is a distributed optimization paradigm that allows training machine learning models across decentralized devices while keeping the data localized. The standard m…