Orloj: Predictably Serving Unpredictable DNNs

Peifeng Yu, Yuqing Qiu, Xin Jin +1

Existing DNN serving solutions can provide tight latency SLOs while maintaining high throughput via careful scheduling of incoming requests, whose execution times are assumed to be…