JITServe: SLO-aware LLM Serving with Imprecise Request Information

Wei Zhang, Zhiyu Wu, Yi Mu +5

The integration of Large Language Models (LLMs) into applications ranging from interactive chatbots to multi-agent systems has introduced a wide spectrum of service-level objective…