3 papers
cs.LG2026
SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling
Weiqiao Shan, Ruixiang Mao, Yuang Li +10
Mixture-of-Experts (MoE) models enable efficient scaling, but training them from scratch remains prohibitively expensive. MoE upcycling mitigates this cost by converting pretrained…
cs.CL2025
NiuTrans.LMT: Toward Inclusive and Scalable Multilingual Machine Translation with LLMs
Yingfeng Luo, Ziqiang Xu, Yuxuan Ouyang +9
Large language models have significantly advanced Multilingual Machine Translation (MMT), yet scaling to many languages while keeping quality robust across directions remains chall…
cs.CL2024
Early Exit Is a Natural Capability in Transformer-based Models: An Empirical Study on Early Exit without Joint Optimization
Weiqiao Shan, Long Meng, Tong Zheng +5
Large language models (LLMs) exhibit exceptional performance across various downstream tasks. However, they encounter limitations due to slow inference speeds stemming from their e…