Contrastive Learning for Many-to-many Multilingual Neural Machine Translation

Abstract

Existing multilingual machine translation approaches mainly focus on English-centric directions, while the non-English directions still lag behind. In this work, we aim to build a many-to-many translation system with an emphasis on the quality of non-English language directions. Our intuition is based on the hypothesis that a universal cross-language representation leads to better multilingual translation performance. To this end, we propose mRASP2, a training method to obtain a single unified multilingual translation model. mRASP2 is empowered by two techniques: a) a contrastive learning scheme to close the gap among representations of different languages, and b) data augmentation on both multiple parallel and monolingual data to further align token representations. For English-centric directions, mRASP2 outperforms existing best unified model and achieves competitive or even better performance than the pre-trained and fine-tuned model mBART on tens of WMT's translation directions. For non-English directions, mRASP2 achieves an improvement of average 10+ BLEU compared with the multilingual Transformer baseline. Code, data and trained models are available at https://github.com/PANXiao1994/mRASP2.

arXiv commentaccepted as long paper in ACL2021

References in corpus (13)

• Multilingual Denoising Pre-training for Neural Machine Translation
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Beyond English-Centric Multilingual Machine Translation
• CLEAR: Contrastive Learning for Sentence Representation
• Multilingual Neural Machine Translation with Knowledge Distillation
• A Teacher-Student Framework for Zero-Resource Neural Machine Translation
• Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
• Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information
• Effective Strategies in Zero-Shot Neural Machine Translation
• Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation
• Enabling Multi-Source Neural Machine Translation By Concatenating Source Sentences In Multiple Languages
• Training Multilingual Machine Translation by Alternately Freezing Language-Specific Encoders-Decoders
• Improving a Multi-Source Neural Machine Translation Model with Corpus Extension for Low-Resource Languages

Cited by in corpus (1)

• Reducing Label Dependency in Human Activity Recognition with Wearables: From Supervised Learning to Novel Weakly Self-Supervised Approaches