4 papers
UniPool: Learning Expert-to-Layer Ownership from Brief Global Access
Minbin Huang, Han Shi, Chuanyang Zheng +5
Most Mixture-of-Experts (MoE) transformers preassign each expert to one layer for the whole of training. We study whether a brief global-access phase can learn a compatible expert-…
Answer-Consistent Chain-of-thought Reinforcement Learning For Multi-modal Large Langauge Models
Minbin Huang, Runhui Huang, Chuanyang Zheng +4
Recent advances in large language models (LLMs) have demonstrated that reinforcement learning with verifiable rewards (RLVR) can significantly enhance reasoning abilities by direct…
Logits-Based Finetuning
Jingyao Li, Senqiao Yang, Sitong Wu +4
In recent years, developing compact and efficient large language models (LLMs) has emerged as a thriving area of research. Traditional Supervised Fine-Tuning (SFT), which relies on…
MiLMo:Minority Multilingual Pre-trained Language Model
Junjie Deng, Hanru Shi, Xinhe Yu +3
Pre-trained language models are trained on large-scale unsupervised data, and they can fine-turn the model only on small-scale labeled datasets, and achieve good results. Multiling…