Showing cs.DCShow all
3 papers · 1 filter
cs.DC2025
Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving
Ke Cheng, Wen Hu, Zhi Wang +3
Large language models (LLMs) iteratively generate text token by token, with memory usage increasing with the length of generated token sequences. Since the request generation lengt…
cs.DC2025
SCOOT: SLO-Oriented Performance Tuning for LLM Inference Engines
Ke Cheng, Zhi Wang, Wen Hu +3
As large language models (LLMs) are gaining increasing popularity across a wide range of web applications, it is of great importance to optimize service-level objectives (SLOs) for…
cs.DC2024
Enabling Efficient Batch Serving for LMaaS via Generation Length Prediction
Ke Cheng, Wen Hu, Zhi Wang +3
Nowadays, large language models (LLMs) are published as a service and can be accessed by various applications via APIs, also known as language-model-as-a-service (LMaaS). Without k…