PiKV: KV Cache Management System for Mixture of Experts

Dong Liu, Yanxuan Yu, Ben Lengerich +1

As large-scale language models continue to scale up in both size and context length, the memory and communication cost of key-value (KV) cache storage has become a major bottleneck…