UniPool: Learning Expert-to-Layer Ownership from Brief Global Access

Minbin Huang, Han Shi, Chuanyang Zheng +5

Most Mixture-of-Experts (MoE) transformers preassign each expert to one layer for the whole of training. We study whether a brief global-access phase can learn a compatible expert-…