4 papers
SPARe: Stacked Parallelism with Adaptive Reordering for Fault-Tolerant LLM Pretraining Systems with 100k+ GPUs
Jin Lee, Zhonghao Chen, Xuhang He +6
In large-scale LLM pre-training systems with 100k+ GPUs, failures become the norm rather than the exception, and restart costs can dominate wall-clock training time. However, exist…
SplitFT: An Adaptive Federated Split Learning System For LLMs Fine-Tuning
Yimeng Shan, Zhaorui Zhang, Sheng Di +3
Federated Split Learning has been identified as an efficient approach to address the computational resource constraints of clients in classical federated learning, while guaranteei…
MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?
Songkai Ma, Zhaorui Zhang, Sheng Di +4
With the widespread application of Mixture of Experts (MoE) reasoning models in the field of LLM learning, efficiently serving MoE models under limited GPU memory constraints has e…
ZCCL: Significantly Improving Collective Communication With Error-Bounded Lossy Compression
Jiajun Huang, Sheng Di, Xiaodong Yu +12
With the ever-increasing computing power of supercomputers and the growing scale of scientific applications, the efficiency of MPI collective communication turns out to be a critic…