Efficient Serving of LLM Applications with Probabilistic Demand Modeling

Yifei Liu, Zuo Gan, Zhenghao Gan +8

Applications based on Large Language Models (LLMs) contains a series of tasks to address real-world problems with boosted capability, which have dynamic demand volumes on diverse b…