most citedLLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

2 citations · 2 across the 2 of their papers we have counts for

collaborators

2 papers

cs.CL2024★ 2 cited

LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

Tong Zhu, Xiaoye Qu, Daize Dong +4

Mixture-of-Experts (MoE) has gained increasing popularity as a promising framework for scaling up large language models (LLMs). However, training MoE from scratch in a large-scale…

cs.CL2024

Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Experts

Tong Zhu, Daize Dong, Xiaoye Qu +3

Mixture-of-Experts (MoE) models have shown remarkable capability in instruction tuning, especially when the number of tasks scales. However, previous methods simply merge all train…