Phrase-Based & Neural Unsupervised Machine Translation

Abstract

Machine translation systems achieve near human-level performance on some languages, yet their effectiveness strongly relies on the availability of large amounts of parallel sentences, which hinders their applicability to the majority of language pairs. This work investigates how to learn to translate when having access to only large monolingual corpora in each language. We propose two model variants, a neural and a phrase-based model. Both versions leverage a careful initialization of the parameters, the denoising effect of language models and automatic generation of parallel data by iterative back-translation. These models are significantly better than methods from the literature, while being simpler and having fewer hyper-parameters. On the widely used WMT'14 English-French and WMT'16 German-English benchmarks, our models respectively obtain 28.1 and 25.2 BLEU points without using a single parallel sentence, outperforming the state of the art by more than 11 BLEU points. On low-resource languages like English-Urdu and English-Romanian, our methods achieve even better results than semi-supervised and supervised approaches leveraging the paucity of available bitexts. Our code for NMT and PBSMT is publicly available.

arXiv commentEMNLP 2018

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Dual Learning for Machine Translation
• Achieving Human Parity on Automatic Chinese to English News Translation
• On Using Monolingual Corpora in Neural Machine Translation
• Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
• A Teacher-Student Framework for Zero-Resource Neural Machine Translation
• Universal Neural Machine Translation for Extremely Low Resource Languages

Cited by in corpus (51)

• Multilingual Denoising Pre-training for Neural Machine Translation
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• Unsupervised Question Answering by Cloze Translation
• Style Transfer as Unsupervised Machine Translation
• Multiple-Attribute Text Style Transfer
• Bilingual Lexicon Induction through Unsupervised Machine Translation
• Zero-Resource Knowledge-Grounded Dialogue Generation
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach
• Self-Paced Learning for Neural Machine Translation
• Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
• Unsupervised Model Selection for Variational Disentangled Representation Learning
• Universal Multimodal Representation for Language Understanding
• Corpus Augmentation by Sentence Segmentation for Low-Resource Neural Machine Translation
• Educating Text Autoencoders: Latent Representation Guidance via Denoising
• Exploration of Neural Machine Translation in Autoformalization of Mathematics in Mizar
• Unsupervised Neural Machine Translation with SMT as Posterior Regularization
• Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
• Revisiting Round-Trip Translation for Quality Estimation
• Rapid Adaptation of Neural Machine Translation to New Languages
• A Simple Baseline to Semi-Supervised Domain Adaptation for Machine Translation
• Latent Translation: Crossing Modalities by Bridging Generative Models
• Emergent Communication Pretraining for Few-Shot Machine Translation
• Automatically Neutralizing Subjective Bias in Text
• Zero-Shot Fine-Grained Style Transfer: Leveraging Distributed Continuous Style Representations to Transfer To Unseen Styles
• From Words to Sentences: A Progressive Learning Approach for Zero-resource Machine Translation with Visual Pivots
• PidginUNMT: Unsupervised Neural Machine Translation from West African Pidgin to English
• Neural Machine Translation: A Review of Methods, Resources, and Tools
• On Synthetic Data for Back Translation
• Neural Semantic Parsing in Low-Resource Settings with Back-Translation and Meta-Learning
• Greedy Search with Probabilistic N-gram Matching for Neural Machine Translation
• Semi-supervised Multimodal Representation Learning through a Global Workspace
• Learning Efficient Lexically-Constrained Neural Machine Translation with External Memory
• AR: Auto-Repair the Synthetic Data for Neural Machine Translation
• Multi-Domain Neural Machine Translation with Word-Level Adaptive Layer-wise Domain Mixing
• Improving Pre-Trained Multilingual Models with Vocabulary Expansion
• The TALP-UPC System for the WMT Similar Language Task: Statistical vs Neural Machine Translation
• Assessing the Bilingual Knowledge Learned by Neural Machine Translation Models
• Paraphrase Generation as Unsupervised Machine Translation
• Multi-word Tokenization for Sequence Compression
• Improving Neural Machine Translation by Bidirectional Training
• Self-Attention with Cross-Lingual Position Representation
• Towards Interlingua Neural Machine Translation
• Knowledge Efficient Deep Learning for Natural Language Processing
• Semi-Supervised Text Simplification with Back-Translation and Asymmetric Denoising Autoencoders
• Unsupervised Neural Dialect Translation with Commonality and Diversity Modeling
• Semi-supervised Text Style Transfer: Cross Projection in Latent Space
• Investigating Code-Mixed Modern Standard Arabic-Egyptian to English Machine Translation
• Cross-Lingual Low-Resource Set-to-Description Retrieval for Global E-Commerce
• Unsupervised Clinical Language Translation
• Language Model-Driven Unsupervised Neural Machine Translation