Tensor Product Attention Is All You Need

Yifan Zhang, Yifeng Liu, Huizhuo Yuan +4

Scaling language models to handle longer input sequences typically necessitates large key-value (KV) caches, resulting in substantial memory overhead during inference. In this pape…