1 citations · 1 across the 8 of their papers we have counts for
Showing 2024Show all
3 papers · 1 filter
cs.LG2024
Wonderful Matrices: Combining for a More Efficient and Effective Foundation Model Architecture
Jingze Shi, Bingheng Wu
In order to make the foundation model more efficient and effective, our idea is combining sequence transformation and state transformation. First, we prove the availability of rota…
cs.CL2024
OTCE: Hybrid SSM and Attention with Cross Domain Mixture of Experts to construct Observer-Thinker-Conceiver-Expresser
Jingze Shi, Ting Xie, Bingheng Wu +2
Recent research has shown that combining Mamba with Transformer architecture, which has selective state space and quadratic self-attention mechanism, outperforms using Mamba or Tra…
cs.LG2024
Wonderful Matrices: More Efficient and Effective Architecture for Language Modeling Tasks
Jingze Shi, Bingheng Wu, Lu He +1
We prove the availability of inner product form position encoding in the state space dual algorithm and study the effectiveness of different position embeddings in the hybrid quadr…