Zeppelin: Balancing Variable-length Workloads in Data Parallel Large Model Training

Chang Chen, Tiancheng Chen, Jiangfei Duan +7

Training large language models (LLMs) with increasingly long and varying sequence lengths introduces severe load imbalance challenges in large-scale data-parallel training. Recent…