InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

Abstract

In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts. The unified view helps us to better understand the existing methods for learning cross-lingual representations. More importantly, inspired by the framework, we propose a new pre-training task based on contrastive learning. Specifically, we regard a bilingual sentence pair as two views of the same meaning and encourage their encoded representations to be more similar than the negative examples. By leveraging both monolingual and parallel corpora, we jointly train the pretext tasks to improve the cross-lingual transferability of pre-trained models. Experimental results on several benchmarks show that our approach achieves considerably better performance. The code and pre-trained models are available at https://aka.ms/infoxlm.

arXiv commentNAACL 2021

References in corpus (14)

• Representation Learning with Contrastive Predictive Coding
• Learning deep representations by mutual information estimation and maximization
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• On the Cross-lingual Transferability of Monolingual Representations
• Multilingual Denoising Pre-training for Neural Machine Translation
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Contrastive Multiview Coding
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• Cross-Lingual Ability of Multilingual BERT: An Empirical Study
• Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
• A Mutual Information Maximization Perspective of Language Representation Learning
• XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation

Cited by in corpus (29)

• Contrastive Representation Learning: A Framework and Review
• mT5: A massively multilingual pre-trained text-to-text transformer
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
• DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
• LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
• Better Fine-Tuning by Reducing Representational Collapse
• MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
• Pre-Trained Models: Past, Present and Future
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
• Contrastive Learning for Cold-Start Recommendation
• On Learning Universal Representations Across Languages
• Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning
• Knowledge-aware Contrastive Molecular Graph Learning
• Contrastive Learning for Robust Android Malware Familial Classification
• Multilingual BERT Post-Pretraining Alignment
• Consistency Regularization for Cross-Lingual Fine-Tuning
• Interest-oriented Universal User Representation via Contrastive Learning
• Empowering News Recommendation with Pre-trained Language Models
• KFCNet: Knowledge Filtering and Contrastive Learning Network for Generative Commonsense Reasoning
• Targeted Adversarial Training for Natural Language Understanding
• Disentangled Contrastive Learning for Learning Robust Textual Representations
• ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
• Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
• Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
• Unbiased Sentence Encoder For Large-Scale Multi-lingual Search Engines
• Nearest Neighbour Few-Shot Learning for Cross-lingual Classification
• Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training