MoE-PHDS: One MoE checkpoint for flexible runtime sparsity

Lauren. A Hannah, Soheil Zibakhsh, Kumari Nishu +4

Sparse Mixtures of Experts (MoEs) are typically trained to operate at a fixed sparsity level, e.g. $k$ in a top-$k$ gating function. This global sparsity level determines an operat…