On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions

Huy Nguyen, Xing Han, Carl Harris +2

With the growing prominence of the Mixture of Experts (MoE) architecture in developing large-scale foundation models, we investigate the Hierarchical Mixture of Experts (HMoE), a s…