On the Cross-lingual Transferability of Monolingual Representations

Abstract

State-of-the-art unsupervised multilingual models (e.g., multilingual BERT) have been shown to generalize in a zero-shot cross-lingual setting. This generalization ability has been attributed to the use of a shared subword vocabulary and joint training across multiple languages giving rise to deep multilingual abstractions. We evaluate this hypothesis by designing an alternative approach that transfers a monolingual model to new languages at the lexical level. More concretely, we first train a transformer-based masked language model on one language, and transfer it to a new language by learning a new embedding matrix with the same masked language modeling objective, freezing parameters of all other layers. This approach does not rely on a shared vocabulary or joint training. However, we show that it is competitive with multilingual BERT on standard cross-lingual classification benchmarks and on a new Cross-lingual Question Answering Dataset (XQuAD). Our results contradict common beliefs of the basis of the generalization ability of multilingual models and suggest that deep monolingual models learn some abstractions that generalize across languages. We also release XQuAD as a more comprehensive cross-lingual benchmark, which comprises 240 paragraphs and 1190 question-answer pairs from SQuAD v1.1 translated into ten languages by professional translators.

ACL 2020

References in corpus (8)

• Cross-lingual Language Model Pretraining
• Assessing BERT's Syntactic Abilities
• Cross-Lingual Ability of Multilingual BERT: An Empirical Study
• Multilingual Alignment of Contextual Word Representations
• Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
• Bilingual Lexicon Induction through Unsupervised Machine Translation
• Emerging Cross-lingual Structure in Pretrained Language Models
• From English To Foreign Languages: Transferring Pre-trained Language Models

Cited by in corpus (68)

• On the Opportunities and Risks of Foundation Models
• Multilingual Denoising Pre-training for Neural Machine Translation
• mT5: A massively multilingual pre-trained text-to-text transformer
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
• CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Pretrained Transformers as Universal Computation Engines
• Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
• Translation Artifacts in Cross-lingual Transfer Learning
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• A Call for More Rigor in Unsupervised Cross-lingual Learning
• Parsing with Multilingual BERT, a Small Corpus, and a Small Treebank
• From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers
• SberQuAD -- Russian Reading Comprehension Dataset: Description and Analysis
• TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages
• WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models
• Emerging Cross-lingual Structure in Pretrained Language Models
• Unsupervised Cross-lingual Adaptation for Sequence Tagging and Beyond
• Resources for Turkish Natural Language Processing: A critical survey
• From English To Foreign Languages: Transferring Pre-trained Language Models
• Pirá: A Bilingual Portuguese-English Dataset for Question-Answering about the Ocean
• FQuAD: French Question Answering Dataset
• mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset
• Are All Languages Created Equal in Multilingual BERT?
• Specializing Multilingual Language Models: An Empirical Study
• RuBQ: A Russian Dataset for Question Answering over Wikidata
• BERT Based Multilingual Machine Comprehension in English and Hindi
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
• LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition
• CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP
• Rethinking embedding coupling in pre-trained language models
• On Learning Universal Representations Across Languages
• Explicit Alignment Objectives for Multilingual Bidirectional Encoders
• LAReQA: Language-agnostic answer retrieval from a multilingual pool
• Filtered Inner Product Projection for Crosslingual Embedding Alignment
• ParsiNLU: A Suite of Language Understanding Challenges for Persian
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
• XOR QA: Cross-lingual Open-Retrieval Question Answering
• LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation
• The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer
• Instance-based Transfer Learning for Multilingual Deep Retrieval
• Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural Networks
• Improving the Robustness of QA Models to Challenge Sets with Variational Question-Answer Pair Generation
• Transferring Monolingual Model to Low-Resource Language: The Case of Tigrinya
• Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation
• CalibreNet: Calibration Networks for Multilingual Sequence Labeling
• Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots
• MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning
• Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models
• Cross-lingual hate speech detection based on multilingual domain-specific word embeddings
• PALI at SemEval-2021 Task 2: Fine-Tune XLM-RoBERTa for Word in Context Disambiguation
• Semantic Search as Extractive Paraphrase Span Detection
• Beyond Offline Mapping: Learning Cross Lingual Word Embeddings through Context Anchoring
• A Multilingual Modeling Method for Span-Extraction Reading Comprehension
• MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
• Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training
• Toward Deconfounding the Influence of Entity Demographics for Question Answering Accuracy
• A cost-benefit analysis of cross-lingual transfer methods
• Cross-language Information Retrieval
• Towards Fully Bilingual Deep Language Modeling
• On the Language-specificity of Multilingual BERT and the Impact of Fine-tuning
• On the Universality of Deep Contextual Language Models
• A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters
• Learning to Generate Task-Specific Adapters from Task Description
• Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation
• Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings