1 citations · 2 across the 8 of their papers we have counts for
8 papers
Eigenvectors of Experts are Training-free Non-collapsing Routers
Giang Do, Hung Le, Truyen Tran
Sparse Mixture of Experts (SMoE) architectures improve the training efficiency of Large Language Models (LLMs) by routing input tokens to a selected subset of specialized experts.…
Do Domain-specific Experts exist in MoE-based LLMs?
Giang Do, Hung Le, Truyen Tran
In the era of Large Language Models (LLMs), the Mixture of Experts (MoE) architecture has emerged as an effective approach for training extremely large models with improved computa…
S2MoE: Robust Sparse Mixture of Experts via Stochastic Learning
Giang Do, Hung Le, Truyen Tran
Sparse Mixture of Experts (SMoE) enables efficient training of large language models by routing input tokens to a select number of experts. However, training SMoE remains challengi…
Rethinking Sparse Mixture of Experts from a Unified Perspective
Giang Do, Hung Le, Truyen Tran
Sparse Mixture of Experts (SMoE) models scale the capacity of models while maintaining constant computational overhead. SMoE methods fall into two categories: Token Choice, which r…
On the Role of Discrete Representation in Sparse Mixture of Experts
Giang Do, Kha Pham, Hung Le +1
Sparse mixture of experts (SMoE) is an effective solution for scaling up model capacity without increasing the computational costs. A crucial component of SMoE is the router, respo…
SimSMoE: Solving Representational Collapse via Similarity Measure
Giang Do, Hung Le, Truyen Tran
Sparse mixture of experts (SMoE) have emerged as an effective approach for scaling large language models while keeping a constant computational cost. Regardless of several notable…
CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition
Quang Pham, Giang Do, Huy Nguyen +8
Sparse mixture of experts (SMoE) offers an appealing solution to scale up the model complexity beyond the mean of increasing the network's depth or width. However, effective traini…
HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts
Giang Do, Khiem Le, Quang Pham +7
By routing input tokens to only a few split experts, Sparse Mixture-of-Experts has enabled efficient training of large language models. Recent findings suggest that fixing the rout…