Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation

Shuaiyi Li, Zhisong Zhang, Yan Wang +5

Block attention, which processes the input as separate blocks that cannot attend to one another, offers significant potential to improve KV cache reuse in long-context scenarios su…