Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System

Yunhua Fang, Rui Xie, Asad Ul Haq +6

Large Language Model (LLM) inference is increasingly constrained by memory bandwidth, with frequent access to the key-value (KV) cache dominating data movement. While attention spa…