Balancing Training for Multilingual Neural Machine Translation

Abstract

When training multilingual machine translation (MT) models that can translate to/from multiple languages, we are faced with imbalanced training sets: some languages have much more training data than others. Standard practice is to up-sample less resourced languages to increase representation, and the degree of up-sampling has a large effect on the overall performance. In this paper, we propose a method that instead automatically learns how to weight training data through a data scorer that is optimized to maximize performance on all test languages. Experiments on two sets of languages under both one-to-many and many-to-one MT settings show our method not only consistently outperforms heuristic baselines in terms of average performance, but also offers flexible control over the performance of which languages are optimized.

Accepted at ACL 2020

References in corpus (4)

• Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation
• Multilingual Neural Machine Translation With Soft Decoupled Encoding
• Choosing Transfer Languages for Cross-Lingual Learning
• Optimizing Data Usage via Differentiable Rewards

Cited by in corpus (7)

• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• Deep Transformers with Latent Depth
• Scaling End-to-End Models for Large-Scale Multilingual ASR
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
• Learning a Multi-Domain Curriculum for Neural Machine Translation
• A Survey on Low-Resource Neural Machine Translation
• Improving Multilingual Translation by Representation and Gradient Regularization