1 citations · 1 across the 6 of their papers we have counts for
4 papers · 1 filter
UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Training
Size Zheng, Xuegui Zheng, Li-wen Chang +1
The exponential growth in Large Language Model (LLM) parameters has transformed model training into an increasingly resource-intensive endeavor. With the stagnation of Moore's Law…
FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Serving
Chia-chi Hsieh, Zan Zong, Xinyang Chen +3
The growing demand for large language models (LLMs) requires serving systems to handle many concurrent requests with diverse service level objectives (SLOs). This exacerbates head-…
A Flexible Programmable Pipeline Parallelism Framework for Efficient DNN Training
Lijuan Jiang, Xingjian Qian, Zhenxiang Ma +4
Pipeline parallelism is an essential distributed parallelism method. Increasingly complex and diverse DNN models necessitate meticulously customized pipeline schedules for performa…
ZeroPP: Unleashing Exceptional Parallelism Efficiency through Tensor-Parallelism-Free Methodology
Ding Tang, Lijuan Jiang, Jiecheng Zhou +5
Large-scale models rely heavily on 3D parallelism for distributed training, which utilizes tensor parallelism (TP) as the intra-operator parallelism to partition model states acros…