2 papers
cs.LG2025
Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations
Ajay Jaiswal, Jianyu Wang, Yixiao Li +6
Sparsely activated Mixture-of-Experts (SMoE) has shown promise in scaling up the learning capacity of neural networks. However, vanilla SMoEs have issues such as expert redundancy…
cs.CL2025
IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining
Yixiao Li, Xianzhi Du, Ajay Jaiswal +4
Recent advancements in large language models have intensified the need for efficient and deployable models within limited inference budgets. Structured pruning pipelines have shown…