WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models

Abstract

Large pretrained language models (LMs) have become the central building block of many NLP applications. Training these models requires ever more computational resources and most of the existing models are trained on English text only. It is exceedingly expensive to train these models in other languages. To alleviate this problem, we introduce a novel method -- called WECHSEL -- to efficiently and effectively transfer pretrained LMs to new languages. WECHSEL can be applied to any model which uses subword-based tokenization and learns an embedding for each subword. The tokenizer of the source model (in English) is replaced with a tokenizer in the target language and token embeddings are initialized such that they are semantically similar to the English tokens by utilizing multilingual static word embeddings covering English and the target language. We use WECHSEL to transfer the English RoBERTa and GPT-2 models to four languages (French, German, Chinese and Swahili). We also study the benefits of our method on very low-resource languages. WECHSEL improves over proposed methods for cross-lingual parameter transfer and outperforms models of comparable size trained from scratch with up to 64x less training effort. Our method makes training large language models for new languages more accessible and less damaging to the environment. We make our code and models publicly available.

arXiv commentNAACL 2022

References in corpus (9)

• Semantics derived automatically from language corpora contain human-like biases
• AraBERT: Transformer-based Model for Arabic Language Understanding
• On the Cross-lingual Transferability of Monolingual Representations
• Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation
• Pre-training via Paraphrasing
• What the [MASK]? Making Sense of Language-Specific BERT Models
• As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages
• From English To Foreign Languages: Transferring Pre-trained Language Models
• Regularization Advantages of Multilingual Neural Language Models for Low Resource Domains

Cited by in corpus (4)

• Transfer to a Low-Resource Language via Close Relatives: The Case Study on Faroese
• Language Models for German Text Simplification: Overcoming Parallel Data Scarcity through Style-specific Pre-training
• FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models
• Architectural Sweet Spots for Modeling Human Label Variation by the Example of Argument Quality: It's Best to Relate Perspectives!