Addressing the Rare Word Problem in Neural Machine Translation

Abstract

Neural Machine Translation (NMT) is a new approach to machine translation that has shown promising results that are comparable to traditional approaches. A significant weakness in conventional NMT systems is their inability to correctly translate very rare words: end-to-end NMTs tend to have relatively small vocabularies with a single unk symbol that represents every possible out-of-vocabulary (OOV) word. In this paper, we propose and implement an effective technique to address this problem. We train an NMT system on data that is augmented by the output of a word alignment algorithm, allowing the NMT system to emit, for each OOV word in the target sentence, the position of its corresponding word in the source sentence. This information is later utilized in a post-processing step that translates every OOV word using a dictionary. Our experiments on the WMT14 English to French translation task show that this method provides a substantial improvement of up to 2.8 BLEU points over an equivalent NMT system that does not use this technique. With 37.5 BLEU points, our NMT system is the first to surpass the best result achieved on a WMT14 contest task.

arXiv commentACL 2015 camera-ready version

References in corpus (1)

• Recurrent Neural Network Regularization

Cited by in corpus (50)

• LSTM: A Search Space Odyssey
• A Neural Conversational Model
• Exploring the Limits of Language Modeling
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• On Using Monolingual Corpora in Neural Machine Translation
• Grammar as a Foreign Language
• SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair
• A Hierarchical Neural Autoencoder for Paragraphs and Documents
• Visualizing and Understanding Neural Models in NLP
• Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networks
• When Are Tree Structures Necessary for Deep Learning of Representations?
• Neural Language Correction with Character-Based Attention
• A Survey of Deep Learning Techniques for Neural Machine Translation
• Efficient Summarization with Read-Again and Copy Mechanism
• Reading Scene Text with Attention Convolutional Sequence Modeling
• Automated Software Vulnerability Assessment with Concept Drift
• Code-Switching for Enhancing NMT with Pre-Specified Translation
• Embedding Word Similarity with Neural Machine Translation
• Deep Learning Based Chatbot Models
• Table-to-text Generation by Structure-aware Seq2seq Learning
• Neural Text Generation: A Practical Guide
• Temporal Learning and Sequence Modeling for a Job Recommender System
• Comparison of Deep learning models on time series forecasting : a case study of Dissolved Oxygen Prediction
• Hindi to English: Transformer-Based Neural Machine Translation
• Memory-augmented Neural Machine Translation
• Differentiable Top-k Operator with Optimal Transport
• Deep Neural Machine Translation with Linear Associative Unit
• Neural Machine Translation with Recurrent Attention Modeling
• TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
• Neural Contextual Conversation Learning with Labeled Question-Answering Pairs
• SMT vs NMT: A Comparison over Hindi & Bengali Simple Sentences
• Exploring the limits of Concurrency in ML Training on Google TPUs
• Log-Linear RNNs: Towards Recurrent Neural Networks with Flexible Prior Knowledge
• Machine Translation : From Statistical to modern Deep-learning practices
• Deep execution monitor for robot assistive tasks
• Intent Recognition in Smart Living Through Deep Recurrent Neural Networks
• An Efficient Character-Level Neural Machine Translation
• Teaching Machines to Converse
• A Convolutional Neural Network for Language-Agnostic Source Code Summarization
• Context Models for OOV Word Translation in Low-Resource Languages
• Radical analysis network for zero-shot learning in printed Chinese character recognition
• A Systematic Comparison of Deep Learning Architectures in an Autonomous Vehicle
• Towards one-shot learning for rare-word translation with external experts
• Sequence-to-Sequence Modeling for Action Identification at High Temporal Resolution
• A Review of Dialogue Systems: From Trained Monkeys to Stochastic Parrots
• Urdu-English Machine Transliteration using Neural Networks
• NEJM-enzh: A Parallel Corpus for English-Chinese Translation in the Biomedical Domain
• Meta-Embedding as Auxiliary Task Regularization
• Translation, Sentiment and Voices: A Computational Model to Translate and Analyse Voices from Real-Time Video Calling
• Multi-task Learning for Low-resource Second Language Acquisition Modeling