Priming: Hybrid State Space Models From Pre-trained Transformers

Aditya Chattopadhyay, Elvis Nunez, Prannay Kaul +6

Hybrid State-Space models combine Attention with recurrent State-Space Model (SSM) layers, balancing eidetic memory from Attention with compressed fading memory from SSMs. This yie…