Adaptive Gating in Mixture-of-Experts based Language Models

Jiamin Li, Qiang Su, Yitao Yang +3

Large language models, such as OpenAI's ChatGPT, have demonstrated exceptional language understanding capabilities in various NLP tasks. Sparsely activated mixture-of-experts (MoE)…