Mixture of Channel Experts: Static Sparse Supports with Input-Adaptive Mixing for Pointwise Projections

Elian Iluk, Gil Ben-Artzi

Mixture-of-Experts (MoE) scales language models by routing each input through a small set of independently parameterized experts. We show that copying this design into convolutiona…