Generating Diverse Translation from Model Distribution with Dropout

Abstract

Despite the improvement of translation quality, neural machine translation (NMT) often suffers from the lack of diversity in its generation. In this paper, we propose to generate diverse translations by deriving a large number of possible models with Bayesian modelling and sampling models from them for inference. The possible models are obtained by applying concrete dropout to the NMT model and each of them has specific confidence for its prediction, which corresponds to a posterior model distribution under specific training data in the principle of Bayesian modeling. With variational inference, the posterior model distribution can be approximated with a variational distribution, from which the final models for inference are sampled. We conducted experiments on Chinese-English and English-German translation tasks and the results shows that our method makes a better trade-off between diversity and accuracy.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Improving neural networks by preventing co-adaptation of feature detectors
• Weight Uncertainty in Neural Networks
• Reducing Transformer Depth on Demand with Structured Dropout
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• Dropout as data augmentation
• deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• Analyzing Uncertainty in Neural Machine Translation
• Sequence to Sequence Mixture Model for Diverse Machine Translation
• Generating Diverse Translation by Manipulating Multi-Head Attention