On the Role of Discrete Representation in Sparse Mixture of Experts

Giang Do, Kha Pham, Hung Le +1

Sparse mixture of experts (SMoE) is an effective solution for scaling up model capacity without increasing the computational costs. A crucial component of SMoE is the router, respo…