Probing Semantic Routing in Large Mixture-of-Expert Models

Matthew Lyle Olson, Neale Ratzlaff, Musashi Hinck +4

In the past year, large (>100B parameter) mixture-of-expert (MoE) models have become increasingly common in the open domain. While their advantages are often framed in terms of eff…