Mixture Models for Diverse Machine Translation: Tricks of the Trade

Abstract

Mixture models trained via EM are among the simplest, most widely used and well understood latent variable models in the machine learning literature. Surprisingly, these models have been hardly explored in text generation applications such as machine translation. In principle, they provide a latent variable to control generation and produce a diverse set of hypotheses. In practice, however, mixture models are prone to degeneracies---often only one component gets trained or the latent variable is simply ignored. We find that disabling dropout noise in responsibility computation is critical to successful training. In addition, the design choices of parameterization, prior distribution, hard versus soft EM and online versus offline assignment can dramatically affect model performance. We develop an evaluation protocol to assess both quality and diversity of generations against multiple references, and provide an extensive empirical study of several mixture model variants. Our analysis shows that certain types of mixture models are more robust and offer the best trade-off between translation quality and diversity compared to variational models and diverse decoding approaches.\footnote{Code to reproduce the results in this paper is available at \url{https://github.com/pytorch/fairseq}}

ICML 2019 camera-ready

Cited by in corpus (16)

• Taming Sparsely Activated Transformer with Stochastic Experts
• FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
• Few-shot Sequence Learning with Transformers
• On the Discrepancy between Density Estimation and Sequence Generation
• A Mixture of $h-1$ Heads is Better than $h$ Heads
• Attention Forcing for Machine Translation
• Goal-driven text descriptions for images
• Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT
• Simple or Complex? Complexity-Controllable Question Generation with Soft Templates and Deep Mixture of Experts Model
• SPMoE: Generate Multiple Pattern-Aware Outputs with Sparse Pattern Mixture of Experts
• multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning
• How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation?
• Mixture of partially linear experts
• WeTS: A Benchmark for Translation Suggestion
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses
• Universal Simultaneous Machine Translation with Mixture-of-Experts Wait-k Policy