Stateful Large Language Model Serving with Pensieve

Lingfan Yu, Jinkun Lin, Jinyang Li

Large Language Models (LLMs) are wildly popular today and it is important to serve them efficiently. Existing LLM serving systems are stateless across requests. Consequently, when…