40 citations · 58 across the 19 of their papers we have counts for
4 papers · 1 filter
Unlocking the Potential of Model Merging for Low-Resource Languages
Mingxu Tao, Chen Zhang, Quzhe Huang +4
Adapting large language models (LLMs) to new languages typically involves continual pre-training (CT) followed by supervised fine-tuning (SFT). However, this CT-then-SFT approach s…
Can Perplexity Reflect Large Language Model's Ability in Long Text Understanding?
Yutong Hu, Quzhe Huang, Mingxu Tao +2
Recent studies have shown that Large Language Models (LLMs) have the potential to process extremely long text. Many works only evaluate LLMs' long-text processing ability on the la…
Harder Tasks Need More Experts: Dynamic Routing in MoE Models
Quzhe Huang, Zhenwei An, Nan Zhuang +7
In this paper, we introduce a novel dynamic expert selection framework for Mixture of Experts (MoE) models, aiming to enhance computational efficiency and model performance by adju…
Teaching Large Language Models an Unseen Language on the Fly
Chen Zhang, Xiao Liu, Jiuheng Lin +1
Existing large language models struggle to support numerous low-resource languages, particularly the extremely low-resource ones, for which there is minimal training data available…