A Call for More Rigor in Unsupervised Cross-lingual Learning

Abstract

We review motivations, definition, approaches, and methodology for unsupervised cross-lingual learning and call for a more rigorous position in each of them. An existing rationale for such research is based on the lack of parallel data for many of the world's languages. However, we argue that a scenario without any parallel data and abundant monolingual data is unrealistic in practice. We also discuss different training signals that have been used in previous work, which depart from the pure unsupervised setting. We then describe common methodological issues in tuning and evaluation of unsupervised cross-lingual models and present best practices. Finally, we provide a unified outlook for different types of research in this area (i.e., cross-lingual word embeddings, deep multilingual pretraining, and unsupervised machine translation) and argue for comparable evaluation of these models.

arXiv commentACL 2020

References in corpus (17)

• Distributed Representations of Words and Phrases and their Compositionality
• Exploiting Similarities among Languages for Machine Translation
• On the Cross-lingual Transferability of Monolingual Representations
• Multilingual Denoising Pre-training for Neural Machine Translation
• A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Unsupervised Statistical Machine Translation
• BilBOWA: Fast Bilingual Distributed Representations without Word Alignments
• Offline bilingual word vectors, orthogonal transformations and the inverted softmax
• An Effective Approach to Unsupervised Machine Translation
• A Teacher-Student Framework for Zero-Resource Neural Machine Translation
• Translation Artifacts in Cross-lingual Transfer Learning
• Bilingual Lexicon Induction through Unsupervised Machine Translation
• Cross-Lingual Machine Reading Comprehension
• Neural Machine Translation with Pivot Languages
• Unsupervised Neural Machine Translation Initialized by Unsupervised Statistical Machine Translation
• Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model

Cited by in corpus (6)

• Learning causal representations for robust domain adaptation
• MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning
• On a Novel Application of Wasserstein-Procrustes for Unsupervised Cross-Lingual Learning
• Self-Learning for Zero Shot Neural Machine Translation
• Predicting the Performance of Multilingual NLP Models
• A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters