Global Simulation-Guided Dynamic Operator Scheduling for Efficient Multi-Tenant Model Serving

Weinan Liu, Zeyuan Ding, Dian Ding +5

Container-granularity scheduling leaves abundant short-lived idle slices within containers unexploited. Reallocating containers is too heavyweight to utilize such fine-grained oppo…