Continuous Semantic Caching for Low-Cost LLM Serving

Baran Atalar, Xutong Liu, Jinhang Zuo +3

As Large Language Models (LLMs) become increasingly popular, caching responses so that they can be reused by users with semantically similar queries has become a vital strategy for…