Load Testing for Machine Learning Model Serving Systems at Scale

Amr S. Abdelfattah, Nakul Tirumalai, Indu Mohanan +4

Machine learning (ML) model serving has become a dominant consumer of GPU infrastructure, yet capacity planning in these systems remains largely ad hoc. Under-provisioning leads to…