Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving

Dev Bali, Soujanya Ponnapalli, Yichuan Wang +3

LLM serving is typically offered as a shared, multi-tenant service, where high-demand workloads from one client can cause latency SLO violations for others. Existing solutions for…