Priority-Aware Preemptive Scheduling for Mixed-Priority Workloads in MoE Inference

Mohammad Siavashi, Faezeh Keshmiri Dindarloo, Dejan Kostic +1

Large Language Models have revolutionized natural language processing, yet serving them efficiently in data centers remains challenging due to mixed workloads comprising latency-se…