ML Inference Scheduling with Predictable Latency

Haidong Zhao, Nikolaos Georgantas

Machine learning (ML) inference serving systems can schedule requests to improve GPU utilization and to meet service level objectives (SLOs) or deadlines. However, improving GPU ut…