Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs

Rongfeng Wang, Shichao Weng, Zhiqiang Wang +4

Mixture-of-Experts (MoE) models route each token to a subset of expert networks, increasing capacity while keeping per-token computation sparse. In many deployed MoEs, the number o…