ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Serving

Haipeng Yuan, Kaining Zheng, Yongshu Bai +5

Current large language model (LLM) inference systems universally deploy ultra-large-scale models using a combination of Tensor Parallelism (TP) and Pipeline Parallelism (PP). Howev…