On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models

William L. Tong, Aryo Lotfi, Emmanuel Abbe +6

State Space Models (SSMs) have emerged as a compelling alternative to Transformers, enabling sequence modeling with constant memory and linear compute. Although SSMs exhibit reason…