On Learning Universal Representations Across Languages
arXiv:2007.15960
Abstract
Recent studies have demonstrated the overwhelming advantage of cross-lingual pre-trained models (PTMs), such as multilingual BERT and XLM, on cross-lingual NLP tasks. However, existing approaches essentially capture the co-occurrence among tokens through involving the masked language model (MLM) objective with token-level cross entropy. In this work, we extend these approaches to learn sentence-level representations and show the effectiveness on cross-lingual understanding and generation. Specifically, we propose a Hierarchical Contrastive Learning (HiCTL) method to (1) learn universal representations for parallel sentences distributed in one or multiple languages and (2) distinguish the semantically-related words from a shared cross-lingual vocabulary for each sentence. We conduct evaluations on two challenging cross-lingual tasks, XTREME and machine translation. Experimental results show that the HiCTL outperforms the state-of-the-art XLM-R by an absolute gain of 4.2% accuracy on the XTREME benchmark as well as achieves substantial improvements on both of the high-resource and low-resource English-to-X translation tasks over strong baselines.
arXiv commentAccepted to ICLR 2021
References in corpus (20)
- Distributed Representations of Words and Phrases and their Compositionality
- Representation Learning with Contrastive Predictive Coding
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Momentum Contrast for Unsupervised Visual Representation Learning
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- On the Cross-lingual Transferability of Monolingual Representations
- Multilingual Denoising Pre-training for Neural Machine Translation
- MASS: Masked Sequence to Sequence Pre-training for Language Generation
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
- A Theoretical Analysis of Contrastive Unsupervised Representation Learning
- Incorporating BERT into Neural Machine Translation
- XNLI: Evaluating Cross-lingual Sentence Representations
- A Mutual Information Maximization Perspective of Language Representation Learning
- InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
- Very Deep Transformers for Neural Machine Translation
- TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages
- Pre-trained Language Model Representations for Language Generation
- On Learning Language-Invariant Representations for Universal Machine Translation