InstCache: A Predictive Cache for LLM Serving

Longwei Zou, Yan Liu, Jiamu Kang +3

The revolutionary capabilities of Large Language Models (LLMs) are attracting rapidly growing popularity and leading to soaring user requests to inference serving systems. Caching…