Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning

Malte Ostendorff, Georg Rehm

Most Transformer language models are primarily pretrained on English text, limiting their use for other languages. As the model sizes grow, the performance gap between English and…