Unsupervised Neural Machine Translation

Abstract

In spite of the recent success of neural machine translation (NMT) in standard benchmarks, the lack of large parallel corpora poses a major practical problem for many language pairs. There have been several proposals to alleviate this issue with, for instance, triangulation and semi-supervised learning techniques, but they still require a strong cross-lingual signal. In this work, we completely remove the need of parallel data and propose a novel method to train an NMT system in a completely unsupervised manner, relying on nothing but monolingual corpora. Our model builds upon the recent work on unsupervised embedding mappings, and consists of a slightly modified attentional encoder-decoder model that can be trained on monolingual corpora alone using a combination of denoising and backtranslation. Despite the simplicity of the approach, our system obtains 15.56 and 10.21 BLEU points in WMT 2014 French-to-English and German-to-English translation. The model can also profit from small parallel corpora, and attains 21.81 and 15.24 points when combined with 100,000 parallel sentences, respectively. Our implementation is released as an open source project.

Published as a conference paper at ICLR 2018

References in corpus (2)

• Sequence to Sequence Learning with Neural Networks
• Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder

Cited by in corpus (49)

• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Multilingual Denoising Pre-training for Neural Machine Translation
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Multimodal Generative Models for Scalable Weakly-Supervised Learning
• MeanSum: A Neural Model for Unsupervised Multi-document Abstractive Summarization
• A Survey of Deep Learning Techniques for Neural Machine Translation
• Pre-training via Paraphrasing
• Zero-Resource Knowledge-Grounded Dialogue Generation
• Unsupervised Text Style Transfer using Language Models as Discriminators
• Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer
• Evaluating prose style transfer with the Bible
• Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach
• Extracting Parallel Sentences with Bidirectional Recurrent Neural Networks to Improve Machine Translation
• Educating Text Autoencoders: Latent Representation Guidance via Denoising
• Meta-Learning for Low-Resource Neural Machine Translation
• EmoGraph: Capturing Emotion Correlations using Graph Networks
• Exploration of Neural Machine Translation in Autoformalization of Mathematics in Mizar
• Unsupervised Neural Machine Translation with SMT as Posterior Regularization
• Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
• Unsupervised Pidgin Text Generation By Pivoting English Data and Self-Training
• On Data-Augmentation and Consistency-Based Semi-Supervised Learning
• Rapid Adaptation of Neural Machine Translation to New Languages
• TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
• What is wrong with style transfer for texts?
• Neural Machine Translation: A Review of Methods, Resources, and Tools
• Language Graph Distillation for Low-Resource Machine Translation
• (Self-Attentive) Autoencoder-based Universal Language Representation for Machine Translation
• Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translation
• PidginUNMT: Unsupervised Neural Machine Translation from West African Pidgin to English
• Recurrent Neural Network-Based Semantic Variational Autoencoder for Sequence-to-Sequence Learning
• Neural Machine Translation for Low-Resourced Indian Languages
• Zero-Resource Cross-Domain Named Entity Recognition
• Deep analytics of atomically-resolved images: manifest and latent features
• Learning to Read by Spelling: Towards Unsupervised Text Recognition
• Unsupervised Dual Paraphrasing for Two-stage Semantic Parsing
• Aligning Vector-spaces with Noisy Supervised Lexicons
• Neural Machine Translation: A Review and Survey
• Sentiment-based Candidate Selection for NMT
• Towards Interlingua Neural Machine Translation
• A Framework for Hierarchical Multilingual Machine Translation
• Paraphrase Generation as Unsupervised Machine Translation
• Neural Data-to-Text Generation with LM-based Text Augmentation
• Explicit Cross-lingual Pre-training for Unsupervised Machine Translation
• Sentence transition matrix: An efficient approach that preserves sentence semantics
• A Hierarchical Neural Network for Sequence-to-Sequences Learning
• Towards Language Agnostic Universal Representations
• Jointly Improving Language Understanding and Generation with Quality-Weighted Weak Supervision of Automatic Labeling
• From Caesar Cipher to Unsupervised Learning: A New Method for Classifier Parameter Estimation
• Language Model-Driven Unsupervised Neural Machine Translation