Sparsity and Superposition in Mixture of Experts

Marmik Chaudhari, Jeremi Nuer, Rome Thorstenson

Mixture of Experts (MoE) models have become central to scaling large language models, yet their mechanistic differences from dense networks remain poorly understood. Previous work…