Showing 2025Show all
2 papers · 1 filter
cs.DC2025
HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap
Wenxiang Lin, Xinglin Pan, Lin Zhang +3
The sparsely activated mixture-of-experts (MoE) transformer has become a common architecture for large language models (LLMs) due to its sparsity, which requires fewer computationa…
cs.LG2025
BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference
Zewen Jin, Shengnan Wang, Jiaan Zhu +5
The Mixture-of-Experts (MoE) structure scales the Transformer-based large language models (LLMs) and improves their performance with only the sub-linear increase in computation res…