Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Training

Zhijun Wang, Jiahuan Li, Hao Zhou +7

Large language models (LLMs) exhibit remarkable multilingual capabilities despite the extreme language imbalance in the pre-training data. In this paper, we closely examine the rea…