Multilingual Alignment of Contextual Word Representations

Abstract

We propose procedures for evaluating and strengthening contextual embedding alignment and show that they are useful in analyzing and improving multilingual BERT. In particular, after our proposed alignment procedure, BERT exhibits significantly improved zero-shot performance on XNLI compared to the base model, remarkably matching pseudo-fully-supervised translate-train models for Bulgarian and Greek. Further, to measure the degree of alignment, we introduce a contextual version of word retrieval and show that it correlates well with downstream zero-shot transfer. Using this word retrieval task, we also analyze BERT and find that it exhibits systematic deficiencies, e.g. worse alignment for open-class parts-of-speech and word pairs written in different scripts, that are corrected by the alignment procedure. These results support contextual alignment as a useful concept for understanding large multilingual pre-trained models.

ICLR 2020

References in corpus (6)

• Distributed Representations of Words and Phrases and their Compositionality
• Cross-lingual Language Model Pretraining
• Exploiting Similarities among Languages for Machine Translation
• Offline bilingual word vectors, orthogonal transformations and the inverted softmax
• Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations
• Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing

Cited by in corpus (40)

• On the Cross-lingual Transferability of Monolingual Representations
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• Parsing with Multilingual BERT, a Small Corpus, and a Small Treebank
• Revisiting the Primacy of English in Zero-shot Cross-lingual Transfer
• From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers
• Unsupervised Cross-lingual Adaptation for Sequence Tagging and Beyond
• Specializing Multilingual Language Models: An Empirical Study
• Inducing Language-Agnostic Multilingual Representations
• USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation
• Word Alignment by Fine-tuning Embeddings on Parallel Corpora
• Explicit Alignment Objectives for Multilingual Bidirectional Encoders
• A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT
• What makes multilingual BERT multilingual?
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
• Bilingual Alignment Pre-Training for Zero-Shot Cross-Lingual Transfer
• Which *BERT? A Survey Organizing Contextualized Encoders
• LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation
• Multilingual BERT Post-Pretraining Alignment
• On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation
• Do Explicit Alignments Robustly Improve Multilingual Encoders?
• Instance-based Transfer Learning for Multilingual Deep Retrieval
• Towards Multi-Sense Cross-Lingual Alignment of Contextual Embeddings
• Combining Static Word Embeddings and Contextual Representations for Bilingual Lexicon Induction
• Cross-lingual Inductive Transfer to Detect Offensive Language
• Looking for Clues of Language in Multilingual BERT to Improve Cross-lingual Generalization
• Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learning
• MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
• Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
• Changes in European Solidarity Before and During COVID-19: Evidence from a Large Crowd- and Expert-Annotated Twitter Dataset
• Cross-Lingual Language Model Meta-Pretraining
• Learning Contextualised Cross-lingual Word Embeddings and Alignments for Extremely Low-Resource Languages Using Parallel Corpora
• Towards Fully Bilingual Deep Language Modeling
• Investigating Post-pretraining Representation Alignment for Cross-Lingual Question Answering
• Improved Multilingual Language Model Pretraining for Social Media Text via Translation Pair Prediction
• A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters
• Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
• Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training
• Multilingual Chart-based Constituency Parse Extraction from Pre-trained Language Models
• DPRK-BERT: The Supreme Language Model
• Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings