Online Scheduling for LLM Inference with KV Cache Constraints

Patrick Jaillet, Jiashuo Jiang, Konstantina Mellou +3

Large Language Model (LLM) inference, where a trained model generates text one word at a time in response to user prompts, is a computationally intensive process requiring efficien…