From English To Foreign Languages: Transferring Pre-trained Language Models

Abstract

Pre-trained models have demonstrated their effectiveness in many downstream natural language processing (NLP) tasks. The availability of multilingual pre-trained models enables zero-shot transfer of NLP tasks from high resource languages to low resource ones. However, recent research in improving pre-trained models focuses heavily on English. While it is possible to train the latest neural architectures for other languages from scratch, it is undesirable due to the required amount of compute. In this work, we tackle the problem of transferring an existing pre-trained model from English to other languages under a limited computational budget. With a single GPU, our approach can obtain a foreign BERT base model within a day and a foreign BERT large within two days. Furthermore, evaluating our models on six languages, we demonstrate that our models are better than multilingual BERT on two zero-shot tasks: natural language inference and dependency parsing.

References in corpus (5)

• On the Cross-lingual Transferability of Monolingual Representations
• From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification
• Cross-Lingual Ability of Multilingual BERT: An Empirical Study
• XNLI: Evaluating Cross-lingual Sentence Representations
• Universal Neural Machine Translation for Extremely Low Resource Languages

Cited by in corpus (7)

• On the Cross-lingual Transferability of Monolingual Representations
• WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models
• A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT
• What makes multilingual BERT multilingual?
• FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models
• On the Universality of Deep Contextual Language Models
• Graph Convolutional Network for Swahili News Classification