Spectral-MoE: Heterogeneous Per-Layer Expert Utilization in Dense-to-MoE Conversion

Zachary Broveak, Christian Kubicka, Thierry Tambe

Large language models impose substantial memory demands and arithmetic costs during inference. Dense-to-mixture-of-experts (MoE) conversion is a model optimization technique that a…