On Using Monolingual Corpora in Neural Machine Translation

Abstract

Recent work on end-to-end neural network-based architectures for machine translation has shown promising results for En-Fr and En-De translation. Arguably, one of the major factors behind this success has been the availability of high quality parallel corpora. In this work, we investigate how to leverage abundant monolingual corpora for neural machine translation. Compared to a phrase-based and hierarchical baseline, we obtain up to $1.96$ BLEU improvement on the low-resource language pair Turkish-English, and $1.59$ BLEU on the focused domain task of Chinese-English chat messages. While our method was initially targeted toward such tasks with less parallel data, we show that it also extends to high resource languages such as Cs-En and De-En where we obtain an improvement of $0.39$ and $0.47$ BLEU scores over the neural machine translation baselines, respectively.

9 pages, 2 figures

References in corpus (5)

• Improving neural networks by preventing co-adaptation of feature detectors
• ADADELTA: An Adaptive Learning Rate Method
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Theano: new features and speed improvements
• Addressing the Rare Word Problem in Neural Machine Translation

Cited by in corpus (10)

• Attention-Based Models for Speech Recognition
• Dual Learning for Machine Translation
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• A Neural Knowledge Language Model
• Neural Machine Translation with Reconstruction
• Towards better decoding and language model integration in sequence to sequence models
• Learning to Decode for Future Success
• Bridging Neural Machine Translation and Bilingual Dictionaries
• Neural Machine Translation with Pivot Languages
• End-to-end attention-based distant speech recognition with Highway LSTM