GoodServe: Towards High-Goodput Serving of Agentic LLM Inferences over Heterogeneous Resources

Boxiao Du, Boning Huangfu, Yizhou Luo +5

Large Language Models (LLMs) play a critical role in emerging agentic applications, where the timely completion of each entire inference is critical. Meanwhile, agentic LLM inferen…