PARD: Enhancing Goodput for Inference Pipeline via Proactive Request Dropping

Zhixin Zhao, Yitao Hu, Simin Chen +8

Modern deep neural network (DNN) applications integrate multiple DNN models into inference pipelines with stringent latency requirements for customized tasks. To mitigate extensive…