First-Token Attraction in Mamba Dynamics

Trinh T. Nguyen, Minh-Khoi Nguyen-Nhat, Duy-Tung Pham +3

We study token dynamics in deep Mamba models, with a particular focus on the attracting role of the first token. Unlike attention mechanisms, where token interactions are governed…