Wonderful Matrices: More Efficient and Effective Architecture for Language Modeling Tasks

Jingze Shi, Bingheng Wu, Lu He +1

We prove the availability of inner product form position encoding in the state space dual algorithm and study the effectiveness of different position embeddings in the hybrid quadr…