SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling

Shashwat Jaiswal, Kunal Jain, Yogesh Simmhan +9

Global cloud service providers handle inference workloads for Large Language Models (LLMs) that span latency-sensitive (e.g., chatbots) and insensitive (e.g., report writing) tasks…