7 papers
Enhancing LLM Training via Spectral Clipping
Xiaowen Jiang, Andrei Semenov, Sebastian U. Stich
While spectral-based optimizers like Muon operate directly on the spectrum of updates, standard adaptive methods such as AdamW do not account for the spectral structure of weights…
Learning When to Adapt
Ali Zindari, Xiaowen Jiang, Rotem Mulayoff +1
Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method, yet its learned correction is static: the same low-rank update is applied to every input. This i…
A practical randomized trust-region method to escape saddle points in high dimension
Radu-Alexandru Dragomir, Xiaowen Jiang, Bonan Sun +1
Without randomization, escaping the saddle points of requires at least pieces of information about (values, gradients, Hessian-ve…
Non-Convex Federated Optimization under Cost-Aware Client Selection
Xiaowen Jiang, Anton Rodomanov, Sebastian U. Stich
Different federated optimization algorithms typically employ distinct client-selection strategies: some methods communicate only with a randomly sampled subset of clients at each r…
FedMuon: Federated Learning with Bias-corrected LMO-based Optimization
Yuki Takezawa, Anastasia Koloskova, Xiaowen Jiang +1
Recently, a new optimization method based on the linear minimization oracle (LMO), called Muon, has been attracting increasing attention since it can train neural networks faster t…
Exploiting Similarity for Computation and Communication-Efficient Decentralized Optimization
Yuki Takezawa, Xiaowen Jiang, Anton Rodomanov +1
Reducing communication complexity is critical for efficient decentralized optimization. The proximal decentralized optimization (PDO) framework is particularly appealing, as method…