Mamba Knockout for Unraveling Factual Information Flow

Nir Endy, Idan Daniel Grosbard, Yuval Ran-Milo +3

This paper investigates the flow of factual information in Mamba State-Space Model (SSM)-based language models. We rely on theoretical and empirical connections to Transformer-base…