Mimetic Initialization Helps State Space Models Learn to Recall

Asher Trockman, Hrayr Harutyunyan, J. Zico Kolter +2

Recent work has shown that state space models such as Mamba are significantly worse than Transformers on recall-based tasks due to the fact that their state size is constant with r…