How multilingual is Multilingual BERT?

Abstract

In this paper, we show that Multilingual BERT (M-BERT), released by Devlin et al. (2018) as a single language model pre-trained from monolingual corpora in 104 languages, is surprisingly good at zero-shot cross-lingual model transfer, in which task-specific annotations in one language are used to fine-tune the model for evaluation in another language. To understand why, we present a large number of probing experiments, showing that transfer is possible even to languages in different scripts, that transfer works best between typologically similar languages, that monolingual corpora can train models for code-switching, and that the model can find translation pairs. From these results, we can conclude that M-BERT does create multilingual representations, but that these representations exhibit systematic deficiencies affecting certain language pairs.

References in corpus (2)

• Cross-lingual Language Model Pretraining
• Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond

Cited by in corpus (68)

• CamemBERT: a Tasty French Language Model
• Multilingual Denoising Pre-training for Neural Machine Translation
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• Multilingual is not enough: BERT for Finnish
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• Low-resource Languages: A Review of Past Work and Future Challenges
• From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers
• Towards Lingua Franca Named Entity Recognition with BERT
• BERT Fine-tuning For Arabic Text Summarization
• Experimental Evaluation of Deep Learning models for Marathi Text Classification
• Deep Transformers with Latent Depth
• A New Dataset for Natural Language Inference from Code-mixed Conversations
• SJ_AJ@DravidianLangTech-EACL2021: Task-Adaptive Pre-Training of Multilingual BERT models for Offensive Language Identification
• Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task
• Automated Concatenation of Embeddings for Structured Prediction
• DICT-MLM: Improved Multilingual Pre-Training using Bilingual Dictionaries
• BERT Based Multilingual Machine Comprehension in English and Hindi
• MTLB-STRUCT @PARSEME 2020: Capturing Unseen Multiword Expressions Using Multi-task Learning and Pre-trained Masked Language Models
• Pre-Trained Models: Past, Present and Future
• Multilingual Question Answering from Formatted Text applied to Conversational Agents
• Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing
• Analysing The Impact Of Linguistic Features On Cross-Lingual Transfer
• Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework
• Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translation
• Do Images really do the Talking? Analysing the significance of Images in Tamil Troll meme classification
• Mono vs Multilingual Transformer-based Models: a Comparison across Several Language Tasks
• Dynamic Data Selection and Weighting for Iterative Back-Translation
• QURIOUS: Question Generation Pretraining for Text Generation
• Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
• Structural Knowledge Distillation: Tractably Distilling Information for Structured Predictor
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
• Enhancing Answer Boundary Detection for Multilingual Machine Reading Comprehension
• Robust Optimization for Multilingual Translation with Imbalanced Data
• LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation
• Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion
• Can Monolingual Pretrained Models Help Cross-Lingual Classification?
• Contextual Lensing of Universal Sentence Representations
• Multilingual BERT Post-Pretraining Alignment
• NLPDove at SemEval-2020 Task 12: Improving Offensive Language Detection with Cross-lingual Transfer
• Fine-tuning BERT-based models for Plant Health Bulletin Classification
• Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model
• kk2018 at SemEval-2020 Task 9: Adversarial Training for Code-Mixing Sentiment Classification
• Language-Independent Tokenisation Rivals Language-Specific Tokenisation for Word Similarity Prediction
• Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation
• Transfer Learning for Multi-lingual Tasks -- a Survey
• UzBERT: pretraining a BERT model for Uzbek
• Towards Multi-Sense Cross-Lingual Alignment of Contextual Embeddings
• Cross-lingual Entity Alignment with Incidental Supervision
• Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural Networks
• Instance-based Transfer Learning for Multilingual Deep Retrieval
• Evaluating Transformer-Based Multilingual Text Classification
• CalibreNet: Calibration Networks for Multilingual Sequence Labeling
• AdvCodeMix: Adversarial Attack on Code-Mixed Data
• Multi-view Subword Regularization
• Probing Contextualized Sentence Representations with Visual Awareness
• Contextual Hate Speech Detection in Code Mixed Text using Transformer Based Approaches
• An Accurate Model for Predicting the (Graded) Effect of Context in Word Similarity Based on Bert
• Language Transfer for Early Warning of Epidemics from Social Media
• To What Degree Can Language Borders Be Blurred In BERT-based Multilingual Spoken Language Understanding?
• Crosslingual Embeddings are Essential in UNMT for Distant Languages: An English to IndoAryan Case Study
• Predicting the Performance of Multilingual NLP Models
• A Data Bootstrapping Recipe for Low Resource Multilingual Relation Classification
• RuSentEval: Linguistic Source, Encoder Force!
• Learning to Represent Bilingual Dictionaries
• Leveraging Adversarial Training in Self-Learning for Cross-Lingual Text Classification
• Rumour Detection via Zero-shot Cross-lingual Transfer Learning
• Ceasing hate withMoH: Hate Speech Detection in Hindi-English Code-Switched Language