Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads

Cunchen Hu, Heyang Huang, Liangliang Xu +9

Transformer-based large language model (LLM) inference serving is now the backbone of many cloud services. LLM inference consists of a prefill phase and a decode phase. However, ex…