RecurFormer: Not All Transformer Heads Need Self-Attention

Ruiqing Yan, Linghan Zheng, Xingbo Du +3

Transformer-based large language models (LLMs) excel in modeling complex language patterns but face significant computational costs during inference, especially with long inputs du…