FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving

Shouwei Gao, Junqi Yin, Feiyi Wang +1

Production LLM serving must simultaneously deliver high throughput, low latency, and sufficient context capacity under non-stationary traffic and mixed request requirements. Data p…