A Queueing Theoretic Perspective on Low-Latency LLM Inference with Variable Token Length

Yuqing Yang, Yuedong Xu, Lei Jiao

Large language models (LLMs) propel the prosperity of interactive AI applications showcased by ChatGPT that demand timely response of inference services. However, LLM inference is…