Mixture of Layers with Hybrid Attention

Ivan Ternovtsii, Yurii Bilak

Standard Mixture-of-Experts (MoE) transformers route tokens to expert subnetworks within each layer, but the layer structure itself remains monolithic. We introduce Mixture of Laye…