Locality-aware Fair Scheduling in LLM Serving

Shiyi Cao, Yichuan Wang, Ziming Mao +10

Large language model (LLM) inference workload dominates a wide variety of modern AI applications, ranging from multi-turn conversation to document analysis. Balancing fairness and…