KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse

Huan Yang, Renji Zhang, Mingzhe Huang +5

Recent advances in long-text understanding have pushed the context length of large language models (LLMs) up to one million tokens. It boosts LLMs's accuracy and reasoning capacity…