Showing cs.LGShow all
3 papers · 1 filter
cs.LG2026
Expert Divergence Learning for MoE-based Language Models
Jiaang Li, Haibin Chen, Langming Liu +9
The Mixture-of-Experts (MoE) architecture is a powerful technique for scaling language models, yet it often suffers from expert homogenization, where experts learn redundant functi…
cs.LG2026
SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursion
Chengting Yu, Xiaobo Shu, Yadao Wang +8
Recursive (looped) Transformers decouple computational depth from parameter depth by repeatedly applying shared layers, providing an explicit architectural primitive for iterative…
cs.LG2025
MeSH: Memory-as-State-Highways for Recursive Transformers
Chengting Yu, Xiaobo Shu, Yadao Wang +8
Recursive transformers reuse parameters and iterate over hidden states multiple times, decoupling compute depth from parameter depth. However, under matched compute, recursive mode…