MeshKV: A Network-on-Chip KV Cache Fabric for Scalable Transformer Decoding Accelerators

Dong Liu, Yanxuan Yu

Autoregressive transformer decoding is constrained by irregular key-value (KV) cache movement on tiled accelerators. Prior compression and DRAM-placement systems still concentrate…