Sequence to Sequence Mixture Model for Diverse Machine Translation

Abstract

Sequence to sequence (SEQ2SEQ) models often lack diversity in their generated translations. This can be attributed to the limitation of SEQ2SEQ models in capturing lexical and syntactic variations in a parallel corpus resulting from different styles, genres, topics, or ambiguity of the translation process. In this paper, we develop a novel sequence to sequence mixture (S2SMIX) model that improves both translation diversity and quality by adopting a committee of specialized translation models rather than a single translation model. Each mixture component selects its own training dataset via optimization of the marginal loglikelihood, which leads to a soft clustering of the parallel corpus. Experiments on four language pairs demonstrate the superiority of our mixture model compared to a SEQ2SEQ baseline with standard or diversity-boosted beam search. Our mixture model uses negligible additional parameters and incurs no extra computation cost during decoding.

arXiv comment11 pages, 5 figures, accepted to CoNLL2018

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Achieving Human Parity on Automatic Chinese to English News Translation
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• Generating Diverse and Accurate Visual Captions by Comparative Adversarial Learning
• Variational Neural Machine Translation
• A Stochastic Decoder for Neural Machine Translation

Cited by in corpus (5)

• Unsupervised Data Augmentation for Consistency Training
• Large-scale Pretraining for Neural Machine Translation with Tens of Billions of Sentence Pairs
• Generating Diverse Translation by Manipulating Multi-Head Attention
• Bayesian Attention Modules
• Generating Diverse Translation from Model Distribution with Dropout