Improving Zero-Shot Translation by Disentangling Positional Information
arXiv:2012.15127
Abstract
Multilingual neural machine translation has shown the capability of directly translating between language pairs unseen in training, i.e. zero-shot translation. Despite being conceptually attractive, it often suffers from low output quality. The difficulty of generalizing to new translation directions suggests the model representations are highly specific to those language pairs seen in training. We demonstrate that a main factor causing the language-specific representations is the positional correspondence to input tokens. We show that this can be easily alleviated by removing residual connections in an encoder layer. With this modification, we gain up to 18.5 BLEU points on zero-shot translation while retaining quality on supervised directions. The improvements are particularly prominent between related languages, where our proposed model outperforms pivot-based translation. Moreover, our approach allows easy integration of new languages, which substantially expands translation coverage. By thorough inspections of the hidden layer outputs, we show that our approach indeed leads to more language-independent representations.
ACL 2021
References in corpus (7)
- Beyond English-Centric Multilingual Machine Translation
- Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
- SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
- The Missing Ingredient in Zero-Shot Neural Machine Translation
- Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
- PMIndia -- A Collection of Parallel Corpora of Languages of India
- Enforcing Encoder-Decoder Modularity in Sequence-to-Sequence Models
Cited by in corpus (5)
- Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
- Language Tags Matter for Zero-Shot Neural Machine Translation
- The Impact of Positional Encodings on Multilingual Compression
- Zero-shot Speech Translation
- Improving Multilingual Translation by Representation and Gradient Regularization