Semantic Scheduling for LLM Inference

Wenyue Hua, Dujian Ding, Yile Gu +4

Conventional operating system scheduling algorithms are largely content-ignorant, making decisions based on factors such as latency or fairness without considering the actual inten…