4 papers
Mixture of Layers with Hybrid Attention
Ivan Ternovtsii, Yurii Bilak
Standard Mixture-of-Experts (MoE) transformers route tokens to expert subnetworks within each layer, but the layer structure itself remains monolithic. We introduce Mixture of Laye…
Geometric Routing Enables Causal Expert Control in Mixture of Experts
Ivan Ternovtsii, Yurii Bilak
Sparse Mixture-of-Experts (MoE) models scale parameters while fixing active computation per token, but the specialization of individual experts remains opaque. In a companion paper…
Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality
Ivan Ternovtsii, Yurii Bilak
Sparse Mixture-of-Experts (MoE) architectures employ increasingly sophisticated routing mechanisms -- learned routers, multi-hop trajectories, token-dependent gating. We ask: does…
Cosine-Similarity Routing with Semantic Anchors for Interpretable Mixture-of-Experts Language Models
Ivan Ternovtsii, Yurii Bilak
Mixture-of-Experts (MoE) models improve efficiency through sparse activation, but their learned gating functions provide limited insight into routing decisions. This work introduce…