Cross-Lingual Ability of Multilingual BERT: An Empirical Study

Abstract

Recent work has exhibited the surprising cross-lingual abilities of multilingual BERT (M-BERT) -- surprising since it is trained without any cross-lingual objective and with no aligned data. In this work, we provide a comprehensive study of the contribution of different components in M-BERT to its cross-lingual ability. We study the impact of linguistic properties of the languages, the architecture of the model, and the learning objectives. The experimental study is done in the context of three typologically different languages -- Spanish, Hindi, and Russian -- and using two conceptually different NLP tasks, textual entailment and named entity recognition. Among our key conclusions is the fact that the lexical overlap between languages plays a negligible role in the cross-lingual success, while the depth of the network is an integral part of it. All our models and implementations can be found on our project page: http://cogcomp.org/page/publication_view/900 .

References in corpus (1)

• Cross-lingual Language Model Pretraining

Cited by in corpus (25)

• Revisiting the Primacy of English in Zero-shot Cross-lingual Transfer
• From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers
• What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological properties
• Semi-automatic Generation of Multilingual Datasets for Stance Detection in Twitter
• Are All Languages Created Equal in Multilingual BERT?
• Code-switched inspired losses for generic spoken dialog representations
• A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT
• NLPDove at SemEval-2020 Task 12: Improving Offensive Language Detection with Cross-lingual Transfer
• Which *BERT? A Survey Organizing Contextualized Encoders
• On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation
• Should we Stop Training More Monolingual Models, and Simply Use Machine Translation Instead?
• Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters
• Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study
• Extending Multilingual BERT to Low-Resource Languages
• Evaluating Transformer-Based Multilingual Text Classification
• Cross-Lingual Language Model Meta-Pretraining
• Language Embeddings for Typology and Cross-lingual Transfer Learning
• Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
• Analyzing the Effects of Reasoning Types on Cross-Lingual Transfer Performance
• Discovering Representation Sprachbund For Multilingual Pre-Training
• Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference
• Nearest Neighbour Few-Shot Learning for Cross-lingual Classification
• A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space
• The Impact of Positional Encodings on Multilingual Compression
• Predicting the Performance of Multilingual NLP Models