MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning
arXiv:2104.07908
Abstract
The combination of multilingual pre-trained representations and cross-lingual transfer learning is one of the most effective methods for building functional NLP systems for low-resource languages. However, for extremely low-resource languages without large-scale monolingual corpora for pre-training or sufficient annotated data for fine-tuning, transfer learning remains an under-studied and challenging task. Moreover, recent work shows that multilingual representations are surprisingly disjoint across languages, bringing additional challenges for transfer onto extremely low-resource languages. In this paper, we propose MetaXL, a meta-learning based framework that learns to transform representations judiciously from auxiliary languages to a target one and brings their representation spaces closer for effective transfer. Extensive experiments on real-world low-resource languages - without access to large-scale monolingual corpora or large amounts of labeled data - for tasks like cross-lingual sentiment analysis and named entity recognition show the effectiveness of our approach. Code for MetaXL is publicly available at github.com/microsoft/MetaXL.
arXiv comment2021 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2021)
References in corpus (8)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- DARTS: Differentiable Architecture Search
- On the Cross-lingual Transferability of Monolingual Representations
- BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
- A Call for More Rigor in Unsupervised Cross-lingual Learning
- Choosing Transfer Languages for Cross-Lingual Learning
- SentiPers: A Sentiment Analysis Corpus for Persian
- Towards Zero-resource Cross-lingual Entity Linking