ReWE: Regressing Word Embeddings for Regularization of Neural Machine Translation Systems

Abstract

Regularization of neural machine translation is still a significant problem, especially in low-resource settings. To mollify this problem, we propose regressing word embeddings (ReWE) as a new regularization technique in a system that is jointly trained to predict the next word in the translation (categorical value) and its word embedding (continuous value). Such a joint training allows the proposed system to learn the distributional properties represented by the word embeddings, empirically improving the generalization to unseen sentences. Experiments over three translation datasets have showed a consistent improvement over a strong baseline, ranging between 0.91 and 2.54 BLEU points, and also a marked improvement over a state-of-the-art system.

arXiv commentAccepted at NAACL-HLT 2019

References in corpus (10)

• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Sequence Level Training with Recurrent Neural Networks
• Pre-Translation for Neural Machine Translation
• Incorporating Discrete Translation Lexicons into Neural Machine Translation
• Guided Alignment Training for Topic-Aware Neural Machine Translation
• Training Neural Machine Translation using Word Embedding-based Loss
• Regularizing Neural Machine Translation by Target-bidirectional Agreement