FairKV: Balancing Per-Head KV Cache for Fast Multi-GPU Inference

Bingzhe Zhao, Ke Cheng, Aomufei Yuan +5

KV cache techniques in Transformer models aim to reduce redundant computations at the expense of substantially increased memory usage, making KV cache compression an important and…