MoRE: Scaling mixture of experts with hardware-aware low-rank routing

Honam Wong, Surbhi Goel, Enric Boix-Adserà

Mixture-of-Experts (MoE) layers are central to frontier language models, and recent architectures push toward more and smaller experts. In this regime, the standard linear router b…