Training Multilingual Machine Translation by Alternately Freezing Language-Specific Encoders-Decoders

Abstract

We propose a modular architecture of language-specific encoder-decoders that constitutes a multilingual machine translation system that can be incrementally extended to new languages without the need for retraining the existing system when adding new languages. Differently from previous works, we simultaneously train $N$ languages in all translation directions by alternately freezing encoder or decoder modules, which indirectly forces the system to train in a common intermediate representation for all languages. Experimental results from multilingual machine translation show that we can successfully train this modular architecture improving on the initial languages while falling slightly behind when adding new languages or doing zero-shot translation. Additional comparison of the quality of sentence representation in the task of natural language inference shows that the alternately freezing training is also beneficial in this direction.

arXiv admin note: text overlap with arXiv:2004.06575

References in corpus (3)

• Adam: A Method for Stochastic Optimization
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• XNLI: Evaluating Cross-lingual Sentence Representations

Cited by in corpus (4)

• Code-switched inspired losses for generic spoken dialog representations
• Contrastive Learning for Many-to-many Multilingual Neural Machine Translation
• Continual Learning in Multilingual NMT via Language-Specific Embeddings
• Improving Zero-shot Neural Machine Translation on Language-specific Encoders-Decoders