A Call for More Rigor in Unsupervised Cross-lingual Learning

Abstract

We review motivations, definition, approaches, and methodology for unsupervised cross-lingual learning and call for a more rigorous position in each of them. An existing rationale for such research is based on the lack of parallel data for many of the world's languages. However, we argue that a scenario without any parallel data and abundant monolingual data is unrealistic in practice. We also discuss different training signals that have been used in previous work, which depart from the pure unsupervised setting. We then describe common methodological issues in tuning and evaluation of unsupervised cross-lingual models and present best practices. Finally, we provide a unified outlook for different types of research in this area (i.e., cross-lingual word embeddings, deep multilingual pretraining, and unsupervised machine translation) and argue for comparable evaluation of these models.

ACL 2020

References in corpus (5)

• Multilingual Denoising Pre-training for Neural Machine Translation
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Offline bilingual word vectors, orthogonal transformations and the inverted softmax
• Translation Artifacts in Cross-lingual Transfer Learning
• Neural Machine Translation with Pivot Languages

Cited by in corpus (6)

• Learning causal representations for robust domain adaptation
• MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning
• On a Novel Application of Wasserstein-Procrustes for Unsupervised Cross-Lingual Learning
• Self-Learning for Zero Shot Neural Machine Translation
• Predicting the Performance of Multilingual NLP Models
• A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters