MixFormer: Linear Transformer with Mixture of Memory Experts

Yu Guo, Lei Duan

State Space Models (SSMs), as a mainstream research direction of linear Transformers, aim to achieve higher efficiency than standard Transformers in long-context modeling. However,…