67 citations · 196 across the 30 of their papers we have counts for
12 papers · 1 filter
Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models
Longteng Zhang, Sen Wu, Shuai Hou +7
Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in…
Reasoning Language Model Inference Serving Unveiled: An Empirical Study
Qi Li, Junpan Wu, Xiang Liu +6
The reasoning large language model (RLLM) has been proven competitive in solving complex reasoning tasks such as mathematics, coding, compared to general LLM. However, the serving…
FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models
Xinglin Pan, Wenxiang Lin, Lin Zhang +5
Recent large language models (LLMs) have tended to leverage sparsity to reduce computations, employing the sparsely activated mixture-of-experts (MoE) technique. MoE introduces fou…
FedImpro: Measuring and Improving Client Update in Federated Learning
Zhenheng Tang, Yonggang Zhang, Shaohuai Shi +4
Federated Learning (FL) models often experience client drift caused by heterogeneous data, where the distribution of data differs across clients. To address this issue, advanced re…
Eva: A General Vectorized Approximation Framework for Second-order Optimization
Lin Zhang, Shaohuai Shi, Bo Li
Second-order optimization algorithms exhibit excellent convergence properties for training deep learning models, but often incur significant computation and memory overheads. This…
Evaluation and Optimization of Gradient Compression for Distributed Deep Learning
Lin Zhang, Longteng Zhang, Shaohuai Shi +2
To accelerate distributed training, many gradient compression methods have been proposed to alleviate the communication bottleneck in synchronous stochastic gradient descent (S-SGD…