Prism Attention: Proposal-Refined Index Sharing Mechanism for Efficient LLMs Inference

Zhenxu Tian, Kebin Liu, Zhengwu Yang +8

Long-context decoding in large language models (LLMs) is increasingly bottlenecked by the memory and computation required to attend over an ever-growing Key-Value (KV) cache at eac…