3 papers
cs.LG2025
UMoE: Unifying Attention and FFN with Shared Experts
Yuanhang Yang, Chaozheng Wang, Jing Li
Sparse Mixture of Experts (MoE) architectures have emerged as a promising approach for scaling Transformer models. While initial works primarily incorporated MoE into feed-forward…
cs.AI2024
Enabling Intelligent Interactions between an Agent and an LLM: A Reinforcement Learning Approach
Bin Hu, Chenyang Zhao, Pu Zhang +4
Large language models (LLMs) encode a vast amount of world knowledge acquired from massive text datasets. Recent studies have demonstrated that LLMs can assist an embodied agent in…
cs.LG2024
XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection
Yuanhang Yang, Shiyi Qi, Wenchao Gu +3
Sparse models, including sparse Mixture-of-Experts (MoE) models, have emerged as an effective approach for scaling Transformer models. However, they often suffer from computational…