3 papers
cs.CL2024
MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing
Hao Zhou, Zhijun Wang, Shujian Huang +6
Large Language Models (LLMs) are often English-centric due to the disproportionate distribution of languages in their pre-training data. Enhancing non-English language capabilities…
cs.CL2024
Multilingual Contrastive Decoding via Language-Agnostic Layers Skipping
Wenhao Zhu, Sizhe Liu, Shujian Huang +3
Decoding by contrasting layers (DoLa), is designed to improve the generation quality of large language models (LLMs) by contrasting the prediction probabilities between an early ex…
cs.CL2024
Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly
Changjiang Gao, Hongda Hu, Peng Hu +3
Despite their strong ability to retrieve knowledge in English, current large language models show imbalance abilities in different languages. Two approaches are proposed to address…