OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration

Youhe Jiang, Fangcheng Fu, Taiyi Wang +2

Serving Large Language Models (LLMs) can benefit immensely from parallelizing both the model and input requests across multiple devices, but incoming workloads exhibit substantial…