MoSKA: Mixture of Shared KV Attention for Efficient Long-Sequence LLM Inference

Myunghyun Rhee, Sookyung Choi, Euiseok Kim +3

The escalating context length in Large Language Models (LLMs) creates a severe performance bottleneck around the Key-Value (KV) cache, whose memory-bound nature leads to significan…