Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function

Tuan Minh Pham, Thinh Cao, Viet Nguyen +3

The sigmoid gate in mixture-of-experts (MoE) models has been empirically shown to outperform the softmax gate across several tasks, ranging from approximating feed-forward networks…