Mutual Information and Diverse Decoding Improve Neural Machine Translation

Abstract

Sequence-to-sequence neural translation models learn semantic and syntactic relations between sentence pairs by optimizing the likelihood of the target given the source, i.e., $p(y|x)$, an objective that ignores other potentially useful sources of information. We introduce an alternative objective function for neural MT that maximizes the mutual information between the source and target sentences, modeling the bi-directional dependency of sources and targets. We implement the model with a simple re-ranking method, and also introduce a decoding algorithm that increases diversity in the N-best list produced by the first pass. Applied to the WMT German/English and French/English tasks, the proposed models offers a consistent performance boost on both standard LSTM and attention-based neural MT architectures.

References in corpus (7)

• Sequence to Sequence Learning with Neural Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• On Using Monolingual Corpora in Neural Machine Translation
• A Hierarchical Neural Autoencoder for Paragraphs and Documents
• A Survey of Available Corpora for Building Data-Driven Dialogue Systems
• A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
• Encoding Source Language with Convolutional Neural Network for Machine Translation

Cited by in corpus (30)

• Diversity in Machine Learning
• Neural Abstractive Text Summarization with Sequence-to-Sequence Models
• Domain Robustness in Neural Machine Translation
• Neural Machine Translation with Reconstruction
• Analyzing Uncertainty in Neural Machine Translation
• Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model
• Neural Language Generation: Formulation, Methods, and Evaluation
• Metasql: A Generate-then-Rank Framework for Natural Language to SQL Translation
• Trading Off Diversity and Quality in Natural Language Generation
• Decoding and Diversity in Machine Translation
• On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation
• Non-Autoregressive Neural Dialogue Generation
• Analysis of diversity-accuracy tradeoff in image captioning
• Making 360${}^{\circ }$ Video Watchable in 2D: Learning Videography for Click Free Viewing
• Black Box Recursive Translations for Molecular Optimization
• Incremental Sampling Without Replacement for Sequence Models
• Mutual Information Maximization for Effective Lip Reading
• Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints
• Towards Unique and Informative Captioning of Images
• Neural Machine Translation: A Review and Survey
• Generating Diverse Story Continuations with Controllable Semantics
• Open-Domain Dialogue Generation Based on Pre-trained Language Models
• There Once Was a Really Bad Poet, It Was Automated but You Didn't Know It
• Goal-driven text descriptions for images
• Single-Queue Decoding for Neural Machine Translation
• REAM$♯$: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generation
• BSDAR: Beam Search Decoding with Attention Reward in Neural Keyphrase Generation
• ConRPG: Paraphrase Generation using Contexts as Regularizer
• Extended Answer and Uncertainty Aware Neural Question Generation
• Latent Part-of-Speech Sequences for Neural Machine Translation