Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patching

Qianli Ma, Zhiqing Tang, Hanshuai Cui +2

Disaggregated serving alleviates memory bottlenecks in Large Language Model (LLM) inference but creates a severe communication bottleneck: transmitting high-dimensional Key-Value (…