Attention, Dynamic Wave and Hybrid Sequence Mixers in Language Models

Dmytro Babarytskyi, Oleksandr Babarytskyi

Large Language Models (LLMs) rely on Multi-Head Attention (MHA) for sequence modeling, but scaling to long contexts is constrained by the quadratic cost of attention, motivating su…