When Does Unsupervised Machine Translation Work?

Abstract

Despite the reported success of unsupervised machine translation (MT), the field has yet to examine the conditions under which these methods succeed, and where they fail. We conduct an extensive empirical evaluation of unsupervised MT using dissimilar language pairs, dissimilar domains, diverse datasets, and authentic low-resource languages. We find that performance rapidly deteriorates when source and target corpora are from different domains, and that random word embedding initialization can dramatically affect downstream translation performance. We additionally find that unsupervised MT performance declines when source and target languages use different scripts, and observe very poor performance on authentic low-resource language pairs. We advocate for extensive empirical evaluation of unsupervised MT systems to highlight failure points and encourage continued research on the most promising paradigms.

arXiv commentWMT20 Camera Ready

References in corpus (12)

• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Exploiting Similarities among Languages for Machine Translation
• A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Unsupervised Statistical Machine Translation
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• An Effective Approach to Unsupervised Machine Translation
• Analyzing the Limitations of Cross-lingual Word Embedding Mappings
• Building Subject-aligned Comparable Corpora and Mining it for Truly Parallel Sentence Pairs
• On the Robustness of Unsupervised and Semi-supervised Cross-lingual Word Embedding Learning
• When and Why is Unsupervised Neural Machine Translation Useless?

Cited by in corpus (8)

• CycleGT: Unsupervised Graph-to-Text and Text-to-Graph Generation via Cycle Training
• USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation
• On Bilingual Lexicon Induction with Large Language Models
• Integrating Unsupervised Data Generation into Self-Supervised Neural Machine Translation for Low-Resource Languages
• Crosslingual Embeddings are Essential in UNMT for Distant Languages: An English to IndoAryan Case Study
• Zero-Shot Language Transfer vs Iterative Back Translation for Unsupervised Machine Translation
• Learning to Pronounce Chinese Without a Pronunciation Dictionary
• Using Document Similarity Methods to create Parallel Datasets for Code Translation