Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT

Abstract

Pretrained contextual representation models (Peters et al., 2018; Devlin et al., 2018) have pushed forward the state-of-the-art on many NLP tasks. A new release of BERT (Devlin, 2018) includes a model simultaneously pretrained on 104 languages with impressive performance for zero-shot cross-lingual transfer on a natural language inference task. This paper explores the broader cross-lingual potential of mBERT (multilingual) as a zero shot language transfer model on 5 NLP tasks covering a total of 39 languages from various language families: NLI, document classification, NER, POS tagging, and dependency parsing. We compare mBERT with the best-published methods for zero-shot cross-lingual transfer and find mBERT competitive on each task. Additionally, we investigate the most effective strategy for utilizing mBERT in this manner, determine to what extent mBERT generalizes away from language specific features, and measure factors that influence cross-lingual transfer.

EMNLP 2019 Camera Ready

References in corpus (6)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• How transferable are features in deep neural networks?
• Cross-lingual Language Model Pretraining
• Word Translation Without Parallel Data
• Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
• The WiLI benchmark dataset for written language identification

Cited by in corpus (13)

• WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
• Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• Cross-lingual Zero- and Few-shot Hate Speech Detection Utilising Frozen Transformer Language Models and AXEL
• Is Multilingual BERT Fluent in Language Generation?
• Multilingual Speech Translation with Efficient Finetuning of Pretrained Models
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
• Can Monolingual Pretrained Models Help Cross-Lingual Classification?
• UniTrans: Unifying Model Transfer and Data Transfer for Cross-Lingual Named Entity Recognition with Unlabeled Data
• Transformer based Multilingual document Embedding model
• Towards Multi-Sense Cross-Lingual Alignment of Contextual Embeddings
• A cost-benefit analysis of cross-lingual transfer methods
• Deriving Disinformation Insights from Geolocalized Twitter Callouts