70 citations · 90 across the 11 of their papers we have counts for
11 papers
Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning
Zichen Tang, Junlin Huang, Rudan Yan +5
Current data compression methods, such as sparsification in Federated Averaging (FedAvg), effectively enhance the communication efficiency of Federated Learning (FL). However, thes…
Scheduling Deep Learning Jobs in Multi-Tenant GPU Clusters via Wise Resource Sharing
Yizhou Luo, Qiang Wang, Shaohuai Shi +4
Deep learning (DL) has demonstrated significant success across diverse fields, leading to the construction of dedicated GPU accelerators within GPU clusters for high-quality traini…
Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules
Xinglin Pan, Wenxiang Lin, Shaohuai Shi +3
Sparsely-activated Mixture-of-Expert (MoE) layers have found practical applications in enlarging the model size of large-scale foundation models, with only a sub-linear increase in…
FedImpro: Measuring and Improving Client Update in Federated Learning
Zhenheng Tang, Yonggang Zhang, Shaohuai Shi +4
Federated Learning (FL) models often experience client drift caused by heterogeneous data, where the distribution of data differs across clients. To address this issue, advanced re…
FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs
Zhenheng Tang, Yuxin Wang, Xin He +8
The rapid growth of memory and computation requirements of large language models (LLMs) has outpaced the development of hardware, hindering people who lack large-scale high-end GPU…
Eva: A General Vectorized Approximation Framework for Second-order Optimization
Lin Zhang, Shaohuai Shi, Bo Li
Second-order optimization algorithms exhibit excellent convergence properties for training deep learning models, but often incur significant computation and memory overheads. This…