Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMs

Hyungwoo Lee, Kihyun Kim, Jinwoo Kim +5

Recent large language models (LLMs) face increasing inference latency as input context length and model size continue to grow. In particular, the retrieval-augmented generation (RA…