SLEM: Spectrum-Aware Latent Expert Merging

Dohyeon Kim, Bedionita Soro, Bruno Andreis +1

Mixture-of-Experts (MoE) models scale model capacity efficiently by activating only a small subset of experts for each input, while still requiring all parameters to be stored in m…