Tail-Replay: Escaping the Curse of Linear Attention in Prefix Caching for Hybrid LLMs

Yirui Liu, Ruoling Qi, Xuaner Wu +2

Hybrid large language models interleave full-attention layers with linear-attention layers to reduce the cost of long-context inference. This structure complicates prefix caching:…