Showing cs.AIShow all
2 papers · 1 filter
cs.AI2026
Full-bandwidth transformer
Xi Wang, Ziyang Cai, Zheng Zhan +5
Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth. Dense attention gives each token broad horizontal acce…
cs.AI2026
The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise
Xi Wang, Soufiane Hayou, Eric Nalisnick
Mixture of Experts (MoEs) are now ubiquitous in large language models, yet the mechanisms behind their "expert specialization" remain poorly understood. We show that, since MoE rou…