gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling

Tianyu Guo, Xianwei Zhang, Jiangsu Du +3

Pipeline parallelism has emerged as a predominant approach for deploying large language models (LLMs) across distributed nodes, owing to its lower communication overhead compared t…