On the Limits of Model Merging for Multilinguality in Pre-Training

Seth Aycock, Fedor Vitiugin, Aleksandr Umnov +2

Endowing models with consistent multilingual performance can be achieved by mixing pre-training data, or post-training approaches such as language-specific model merging. In this w…