SANTA++: Sampling Attention through Representative Keys

Kyle Lee, Christian Z. Pratt, Ruoyu Fang +4

Attention often concentrates on a small subset of tokens in the context, but which subset matters changes from one query to the next. To exploit this changing structure, we introdu…