Balancing Computation Load and Representation Expressivity in Parallel Hybrid Neural Networks

Mohammad Mahdi Moradi, Walid Ahmed, Shuangyue Wen +3

Attention and State-Space Models (SSMs) when combined in a hybrid network in sequence or in parallel provide complementary strengths. In a hybrid sequential pipeline they alternate…