1 citations · 2 across the 8 of their papers we have counts for
4 papers · 1 filter
Eigenvectors of Experts are Training-free Non-collapsing Routers
Giang Do, Hung Le, Truyen Tran
Sparse Mixture of Experts (SMoE) architectures improve the training efficiency of Large Language Models (LLMs) by routing input tokens to a selected subset of specialized experts.…
On the Role of Discrete Representation in Sparse Mixture of Experts
Giang Do, Kha Pham, Hung Le +1
Sparse mixture of experts (SMoE) is an effective solution for scaling up model capacity without increasing the computational costs. A crucial component of SMoE is the router, respo…
CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition
Quang Pham, Giang Do, Huy Nguyen +8
Sparse mixture of experts (SMoE) offers an appealing solution to scale up the model complexity beyond the mean of increasing the network's depth or width. However, effective traini…
HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts
Giang Do, Khiem Le, Quang Pham +7
By routing input tokens to only a few split experts, Sparse Mixture-of-Experts has enabled efficient training of large language models. Recent findings suggest that fixing the rout…