A Tale of Two Scales: Reconciling Horizontal and Vertical Scaling for Inference Serving Systems

Kamran Razavi, Mehran Salmani, Max Mühlhäuser +2

Inference serving is of great importance in deploying machine learning models in real-world applications, ensuring efficient processing and quick responses to inference requests. H…