Cross-Lingual Ability of Multilingual BERT: An Empirical Study

Abstract

Recent work has exhibited the surprising cross-lingual abilities of multilingual BERT (M-BERT) -- surprising since it is trained without any cross-lingual objective and with no aligned data. In this work, we provide a comprehensive study of the contribution of different components in M-BERT to its cross-lingual ability. We study the impact of linguistic properties of the languages, the architecture of the model, and the learning objectives. The experimental study is done in the context of three typologically different languages -- Spanish, Hindi, and Russian -- and using two conceptually different NLP tasks, textual entailment and named entity recognition. Among our key conclusions is the fact that the lexical overlap between languages plays a negligible role in the cross-lingual success, while the depth of the network is an integral part of it. All our models and implementations can be found on our project page: http://cogcomp.org/page/publication_view/900 .

References in corpus (1)

• Cross-lingual Language Model Pretraining

Cited by in corpus (55)

• Pre-trained Models for Natural Language Processing: A Survey
• On the Cross-lingual Transferability of Monolingual Representations
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• Cross-lingual Retrieval for Iterative Self-Supervised Training
• Parsing with Multilingual BERT, a Small Corpus, and a Small Treebank
• Revisiting the Primacy of English in Zero-shot Cross-lingual Transfer
• From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers
• What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological properties
• Unsupervised Cross-lingual Adaptation for Sequence Tagging and Beyond
• Emerging Cross-lingual Structure in Pretrained Language Models
• Semi-automatic Generation of Multilingual Datasets for Stance Detection in Twitter
• From English To Foreign Languages: Transferring Pre-trained Language Models
• Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task
• Specializing Multilingual Language Models: An Empirical Study
• Are All Languages Created Equal in Multilingual BERT?
• SOLD: Sinhala Offensive Language Dataset
• Code-switched inspired losses for generic spoken dialog representations
• Rethinking embedding coupling in pre-trained language models
• Explicit Alignment Objectives for Multilingual Bidirectional Encoders
• A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT
• What makes multilingual BERT multilingual?
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
• Which *BERT? A Survey Organizing Contextualized Encoders
• NLPDove at SemEval-2020 Task 12: Improving Offensive Language Detection with Cross-lingual Transfer
• Multilingual BERT Post-Pretraining Alignment
• On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation
• Should we Stop Training More Monolingual Models, and Simply Use Machine Translation Instead?
• Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters
• Extending Multilingual BERT to Low-Resource Languages
• Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study
• Instance-based Transfer Learning for Multilingual Deep Retrieval
• Evaluating Transformer-Based Multilingual Text Classification
• Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots
• More Embeddings, Better Sequence Labelers?
• An Empirical Study of Pre-trained Transformers for Arabic Information Extraction
• CLIPScore: A Reference-free Evaluation Metric for Image Captioning
• Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
• Language Embeddings for Typology and Cross-lingual Transfer Learning
• Detecting Foodborne Illness Complaints in Multiple Languages Using English Annotations Only
• MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
• Cross-Lingual Language Model Meta-Pretraining
• Building Low-Resource NER Models Using Non-Speaker Annotation
• A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space
• On the Universality of Deep Contextual Language Models
• "Average" Approximates "First Principal Component"? An Empirical Analysis on Representations from Neural Language Models
• Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training
• Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference
• Towards Fully Bilingual Deep Language Modeling
• Discovering Representation Sprachbund For Multilingual Pre-Training
• Nearest Neighbour Few-Shot Learning for Cross-lingual Classification
• The Impact of Positional Encodings on Multilingual Compression
• On the Language-specificity of Multilingual BERT and the Impact of Fine-tuning
• Analyzing the Effects of Reasoning Types on Cross-Lingual Transfer Performance
• Predicting the Performance of Multilingual NLP Models