On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
arXiv:2010.03017
Abstract
Modern multilingual models are trained on concatenated text from multiple languages in hopes of conferring benefits to each (positive transfer), with the most pronounced benefits accruing to low-resource languages. However, recent work has shown that this approach can degrade performance on high-resource languages, a phenomenon known as negative interference. In this paper, we present the first systematic study of negative interference. We show that, contrary to previous belief, negative interference also impacts low-resource languages. While parameters are maximally shared to learn language-universal structures, we demonstrate that language-specific parameters do exist in multilingual models and they are a potential cause of negative interference. Motivated by these observations, we also present a meta-learning algorithm that obtains better cross-lingual transferability and alleviates negative interference, by adding language-specific layers as meta-parameters and training them in a manner that explicitly improves shared layers' generalization on all languages. Overall, our results show that negative interference is more common than previously known, suggesting new directions for improving multilingual representations.
arXiv commentPublished as a main conference paper at EMNLP 2020
References in corpus (23)
- Adam: A Method for Stochastic Optimization
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- An Overview of Multi-Task Learning in Deep Neural Networks
- Cross-lingual Language Model Pretraining
- On the Cross-lingual Transferability of Monolingual Representations
- Learning multiple visual domains with residual adapters
- Similarity of Neural Network Representations Revisited
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
- SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
- Cross-Lingual Ability of Multilingual BERT: An Empirical Study
- Multilingual Alignment of Contextual Word Representations
- XNLI: Evaluating Cross-lingual Sentence Representations
- Parameter-Efficient Transfer Learning for NLP
- How multilingual is Multilingual BERT?
- Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
- Gradient Surgery for Multi-Task Learning
- Online Learning Rate Adaptation with Hypergradient Descent
- Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT
- TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages
- Emerging Cross-lingual Structure in Pretrained Language Models
- Simple, Scalable Adaptation for Neural Machine Translation
- Balancing Training for Multilingual Neural Machine Translation