2 papers
cs.DC2026
MemFine: Memory-Aware Fine-Grained Scheduling for MoE Training
Lu Zhao, Rong Shi, Shaoqing Zhang +21
The training of large-scale Mixture of Experts (MoE) models faces a critical memory bottleneck due to severe load imbalance caused by dynamic token routing. This imbalance leads to…
cs.DC2025
MoFa: A Unified Performance Modeling Framework for LLM Pretraining
Lu Zhao, Rong Shi, Shaoqing Zhang +14
The exponential growth in LLM scales, with parameters soaring from billions to trillions, has necessitated distributed pretraining across large clusters comprising thousands to ten…