Towards High-Goodput LLM Serving with Prefill-decode Multiplexing

Yukang Chen, Weihao Cui, Han Zhao +7

Large Language Model (LLM) serving must meet stringent Service Level Objectives (SLOs) for both the prefill and decode phases. Some existing solutions disaggregate the two phases,…