Attention to Mamba: A Recipe for Cross-Architecture Distillation

Abhinav Moudgil, Ningyuan Huang, Eeshan Gunesh Dhekane +3

State Space Models (SSMs) such as Mamba have become a popular alternative to Transformer models, due to their reduced memory consumption and higher throughput at generation compare…