3 papers
cs.DC2026
PROBE: Co-Balancing Computation and Communication in MoE Inference via Real-Time Predictive Prefetching
Qianchao Zhu, Xucheng Ye, Yuliang Liu +2
Mixture-of-Experts models have become a dominant architecture for scaling Large Language Models by activating only a sparse subset of experts per token. However, latency-critical M…
cs.AI2025
SlimPack: Fine-Grained Asymmetric Packing for Balanced and Efficient Variable-Length LLM Training
Yuliang Liu, Guohao Wu, Shenglong Zhang +4
The efficient distributed training of Large Language Models (LLMs) is severely hampered by the extreme variance in context lengths. This data heterogeneity, amplified by convention…
cs.LG2025
SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Training
Zhouyang Li, Yuliang Liu, Wei Zhang +4
Pipeline Parallelism (PP) serves as a crucial technique for training Large Language Models (LLMs), owing to its capability to alleviate memory pressure from model states with relat…