Xiaodong Zheng

collaborators

2 papers

cs.LG2026

From $O (mn)$ to $O (r^{2})$ : Two-Sided Low-Rank Communication for Adam in Distributed Training with Memory Efficiency

Sizhe Dang, Jiaqi Shao, Xiaodong Zheng +3

As foundation models continue to scale, pretraining increasingly relies on data-parallel distributed optimization, making bandwidth-limited gradient synchronization a key bottlenec…

cs.LG2025

FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed

Sizhe Dang, Yangyang Guo, Yanjun Zhao +4

Fine-tuning large language models (LLMs) often faces GPU memory bottlenecks: the backward pass of first-order optimizers like Adam increases memory usage to more than 10 times the…

From O(mn) to O(r2): Two-Sided Low-Rank Communication for Adam in Distributed Training with Memory Efficiency

FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed

From $O (mn)$ to $O (r^{2})$ : Two-Sided Low-Rank Communication for Adam in Distributed Training with Memory Efficiency