12 citations · 24 across the 7 of their papers we have counts for
4 papers · 1 filter
Eva: A General Vectorized Approximation Framework for Second-order Optimization
Lin Zhang, Shaohuai Shi, Bo Li
Second-order optimization algorithms exhibit excellent convergence properties for training deep learning models, but often incur significant computation and memory overheads. This…
LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning
Longteng Zhang, Lin Zhang, Shaohuai Shi +2
Fine-tuning large language models (LLMs) is crucial for improving their performance on downstream tasks, but full-parameter fine-tuning (Full-FT) is computationally expensive and m…
Evaluation and Optimization of Gradient Compression for Distributed Deep Learning
Lin Zhang, Longteng Zhang, Shaohuai Shi +2
To accelerate distributed training, many gradient compression methods have been proposed to alleviate the communication bottleneck in synchronous stochastic gradient descent (S-SGD…
DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipelining
Lin Zhang, Shaohuai Shi, Xiaowen Chu +3
Communication scheduling has been shown to be effective in accelerating distributed training, which enables all-reduce communications to be overlapped with backpropagation computat…