Robustness of Mixtures of Experts to Feature Noise

Dong Sun, Rahul Nittala, Rebekka Burkholz

Despite their practical success, it remains unclear why Mixture of Experts (MoE) models can outperform dense networks beyond sheer parameter scaling. We study an iso-parameter regi…