On the Spatial Structure of Mixture-of-Experts in Transformers

Daniel Bershatsky, Ivan Oseledets

A common assumption is that MoE routers primarily leverage semantic features for expert selection. However, our study challenges this notion by demonstrating that positional token…