Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Models

Bo Wang, Junzhuo Li, Hong Chen +3

Mixture-of-Experts (MoE) architectures decouple model capacity from per-token computation, enabling scaling beyond the computational limits imposed by dense scaling laws. Yet how M…