SCOOT: SLO-Oriented Performance Tuning for LLM Inference Engines

Ke Cheng, Zhi Wang, Wen Hu +3

As large language models (LLMs) are gaining increasing popularity across a wide range of web applications, it is of great importance to optimize service-level objectives (SLOs) for…