Can Monolingual Pretrained Models Help Cross-Lingual Classification?
arXiv:1911.03913
Abstract
Multilingual pretrained language models (such as multilingual BERT) have achieved impressive results for cross-lingual transfer. However, due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. In this work, we present two approaches to improve zero-shot cross-lingual classification, by transferring the knowledge from monolingual pretrained models to multilingual ones. Experimental results on two cross-lingual classification benchmarks show that our methods outperform vanilla multilingual fine-tuning.
5 pages
References in corpus (9)
- Distilling the Knowledge in a Neural Network
- Cross-lingual Language Model Pretraining
- Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
- XNLI: Evaluating Cross-lingual Sentence Representations
- How multilingual is Multilingual BERT?
- Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT
- Cross-lingual Distillation for Text Classification
- MultiFiT: Efficient Multi-lingual Language Model Fine-tuning
- Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER