Do Domain-specific Experts exist in MoE-based LLMs?

Giang Do, Hung Le, Truyen Tran

In the era of Large Language Models (LLMs), the Mixture of Experts (MoE) architecture has emerged as an effective approach for training extremely large models with improved computa…