Multilingual NMT with a language-independent attention bridge

Abstract

In this paper, we propose a multilingual encoder-decoder architecture capable of obtaining multilingual sentence representations by means of incorporating an intermediate {\em attention bridge} that is shared across all languages. That is, we train the model with language-specific encoders and decoders that are connected via self-attention with a shared layer that we call attention bridge. This layer exploits the semantics from each language for performing translation and develops into a language-independent meaning representation that can efficiently be used for transfer learning. We present a new framework for the efficient development of multilingual NMT using this model and scheduled training. We have tested the approach in a systematic way with a multi-parallel data set. We show that the model achieves substantial improvements over strong bilingual models and that it also works well for zero-shot translation, which demonstrates its ability of abstraction and transfer learning.

References in corpus (14)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Convolutional Sequence to Sequence Learning
• A Structured Self-attentive Sentence Embedding
• SentEval: An Evaluation Toolkit for Universal Sentence Representations
• What you can cram into a single vector: Probing sentence embeddings for linguistic properties
• Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
• A Teacher-Student Framework for Zero-Resource Neural Machine Translation
• Multilingual Neural Machine Translation with Task-Specific Attention
• Enhancing Sentence Embedding with Generalized Pooling
• Improving Zero-Shot Translation of Low-Resource Languages
• A neural interlingua for multilingual machine translation
• Contextual Parameter Generation for Universal Neural Machine Translation

Cited by in corpus (10)

• Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• A Comprehensive Survey of Multilingual Neural Machine Translation
• HLT-MT: High-resource Language-specific Training for Multilingual Neural Machine Translation
• Learning Shared Semantic Space for Speech-to-Text Translation
• A Brief Survey of Multilingual Neural Machine Translation
• Revisiting Language Encoding in Learning Multilingual Representations
• Exploring Pair-Wise NMT for Indian Languages
• Importance-based Neuron Allocation for Multilingual Neural Machine Translation
• Improving Zero-shot Neural Machine Translation on Language-specific Encoders-Decoders