Multi-Turn Semantic Caching for Low-Cost LLM Serving

Baran Atalar, Selly Beyene, Xutong Liu +1

Semantic caching lowers the cost of serving large language models (LLMs) by reusing stored responses for similar requests. In multi-turn conversations, whether a cached response is…