Cross-lingual Language Model Pretraining

Abstract

Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding. In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining. We propose two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual data, and one supervised that leverages parallel data with a new cross-lingual language model objective. We obtain state-of-the-art results on cross-lingual classification, unsupervised and supervised machine translation. On XNLI, our approach pushes the state of the art by an absolute gain of 4.9% accuracy. On unsupervised machine translation, we obtain 34.3 BLEU on WMT'16 German-English, improving the previous state of the art by more than 9 BLEU. On supervised machine translation, we obtain a new state of the art of 38.5 BLEU on WMT'16 Romanian-English, outperforming the previous best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Exploiting Similarities among Languages for Machine Translation
• Exploring the Limits of Language Modeling
• A large annotated corpus for learning natural language inference
• Massively Multilingual Word Embeddings
• XNLI: Evaluating Cross-lingual Sentence Representations
• Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
• Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation
• Multilingual Models for Compositional Distributed Semantics
• Unsupervised Cross-lingual Word Embedding by Multilingual Neural Language Models

Cited by in corpus (240)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• BERTScore: Evaluating Text Generation with BERT
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Pre-trained Models for Natural Language Processing: A Survey
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• ERNIE: Enhanced Representation through Knowledge Integration
• CamemBERT: a Tasty French Language Model
• On the Cross-lingual Transferability of Monolingual Representations
• Multilingual Denoising Pre-training for Neural Machine Translation
• mT5: A massively multilingual pre-trained text-to-text transformer
• TAPAS: Weakly Supervised Table Parsing via Pre-training
• Q8BERT: Quantized 8Bit BERT
• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
• Evaluating Word Embedding Models: Methods and Experimental Results
• CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• A Transformer-based approach to Irony and Sarcasm detection
• Reducing Transformer Depth on Demand with Structured Dropout
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• Controllable Protein Design with Language Models
• Transformers in Healthcare: A Survey
• Cross-Lingual Ability of Multilingual BERT: An Empirical Study
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• GraphCodeBERT: Pre-training Code Representations with Data Flow
• MuRIL: Multilingual Representations for Indian Languages
• Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
• Multilingual Alignment of Contextual Word Representations
• VideoBERT: A Joint Model for Video and Language Representation Learning
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• Information Retrieval: Recent Advances and Beyond
• How multilingual is Multilingual BERT?
• Unsupervised Question Answering by Cloze Translation
• An Effective Approach to Unsupervised Machine Translation
• CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
• Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings
• Multilingual is not enough: BERT for Finnish
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• Training with Quantization Noise for Extreme Model Compression
• Entailment as Few-Shot Learner
• Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
• Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
• DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Knowledge Graph informed Fake News Classification via Heterogeneous Representation Ensembles
• Pre-training via Paraphrasing
• GREEK-BERT: The Greeks visiting Sesame Street
• A Mutual Information Maximization Perspective of Language Representation Learning
• Representation Learning for Natural Language Processing
• PyThaiNLP: Thai Natural Language Processing in Python
• Recommending Metamodel Concepts during Modeling Activities with Pre-Trained Language Models
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
• RecGURU: Adversarial Learning of Generalized User Representations for Cross-Domain Recommendation
• MisRoBÆRTa: Transformers versus Misinformation
• Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems
• Unsupervised Translation of Programming Languages
• XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering
• Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT
• HPC-GPT: Integrating Large Language Model for High-Performance Computing
• Cross-lingual Zero- and Few-shot Hate Speech Detection Utilising Frozen Transformer Language Models and AXEL
• Towards Offensive Language Identification for Tamil Code-Mixed YouTube Comments and Posts
• Self-supervised Consensus Representation Learning for Attributed Graph
• Exploiting BERT for End-to-End Aspect-based Sentiment Analysis
• Gmail Smart Compose: Real-Time Assisted Writing
• A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models
• Incorporating BERT into Parallel Sequence Decoding with Adapters
• On the Effectiveness of Transfer Learning for Code Search
• Diet Code Is Healthy: Simplifying Programs for Pre-trained Models of Code
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• A Focus on Neural Machine Translation for African Languages
• Transfer Language Selection for Zero-Shot Cross-Lingual Abusive Language Detection
• 12-in-1: Multi-Task Vision and Language Representation Learning
• Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models
• Zero-shot cross-lingual transfer language selection using linguistic similarity
• Disentangling Syntax and Semantics in the Brain with Deep Networks
• IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP
• SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answering
• Multilingual Speech Translation with Efficient Finetuning of Pretrained Models
• Improving Word Translation via Two-Stage Contrastive Learning
• Unified Abstract Syntax Tree Representation Learning for Cross-Language Program Classification
• Tree-structured Attention with Hierarchical Accumulation
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
• Emerging Cross-lingual Structure in Pretrained Language Models
• Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model
• DocBank: A Benchmark Dataset for Document Layout Analysis
• Knowledge Distillation from Internal Representations
• Experimental Evaluation of Deep Learning models for Marathi Text Classification
• A Comprehensive Survey of Multilingual Neural Machine Translation
• Encoder-Agnostic Adaptation for Conditional Language Generation
• CrossNER: Evaluating Cross-Domain Named Entity Recognition
• Contextual Word Representations: A Contextual Introduction
• Distilling Knowledge Learned in BERT for Text Generation
• Teaching a New Dog Old Tricks: Resurrecting Multilingual Retrieval Using Zero-shot Learning
• Exploration of Neural Machine Translation in Autoformalization of Mathematics in Mizar
• Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization
• Joint Source-Target Self Attention with Locality Constraints
• BERT-like Pre-training for Symbolic Piano Music Classification Tasks
• Named entity recognition architecture combining contextual and global features
• DOBF: A Deobfuscation Pre-Training Objective for Programming Languages
• Specializing Multilingual Language Models: An Empirical Study
• The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives
• Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabularies
• KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understanding
• UZH_CLyp at SemEval-2023 Task 9: Head-First Fine-Tuning and ChatGPT Data Generation for Cross-Lingual Learning in Tweet Intimacy Prediction
• NewsEmbed: Modeling News through Pre-trained Document Representations
• Kencorpus: A Kenyan Language Corpus of Swahili, Dholuo and Luhya for Natural Language Processing Tasks
• Cross-Lingual Named Entity Recognition Using Parallel Corpus: A New Approach Using XLM-RoBERTa Alignment
• Zero-Shot Paraphrase Generation with Multilingual Language Models
• Can Unconditional Language Models Recover Arbitrary Sentences?
• Finding Experts in Transformer Models
• Bridging the domain gap in cross-lingual document classification
• Attention-Informed Mixed-Language Training for Zero-shot Cross-lingual Task-oriented Dialogue Systems
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
• Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations
• Multilingual Question Answering from Formatted Text applied to Conversational Agents
• Deep Learning Transformer Architecture for Named Entity Recognition on Low Resourced Languages: State of the art results
• Robust Cross-lingual Embeddings from Parallel Sentences
• Aspect and Opinion Term Extraction for Hotel Reviews using Transfer Learning and Auxiliary Labels
• From Handcrafted Features to LLMs: A Brief Survey for Machine Translation Quality Estimation
• Code-switched inspired losses for generic spoken dialog representations
• Emergent Communication Pretraining for Few-Shot Machine Translation
• CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP
• Semi-Supervised Models via Data Augmentationfor Classifying Interactive Affective Responses
• Acquiring Knowledge from Pre-trained Model to Neural Machine Translation
• SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
• Linguistic Dependencies and Statistical Dependence
• LAnoBERT: System Log Anomaly Detection based on BERT Masked Language Model
• Give your Text Representation Models some Love: the Case for Basque
• Bilingual Dictionary Based Neural Machine Translation without Using Parallel Sentences
• LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning
• Mixed-Lingual Pre-training for Cross-lingual Summarization
• A Financial Service Chatbot based on Deep Bidirectional Transformers
• Que2Engage: Embedding-based Retrieval for Relevant and Engaging Products at Facebook Marketplace
• Literature Retrieval for Precision Medicine with Neural Matching and Faceted Summarization
• Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translation
• Large-scale Pretraining for Neural Machine Translation with Tens of Billions of Sentence Pairs
• Transform and Tell: Entity-Aware News Image Captioning
• Neural Machine Translation: A Review of Methods, Resources, and Tools
• ParaLaw Nets -- Cross-lingual Sentence-level Pretraining for Legal Text Processing
• LAMPAT: Low-Rank Adaption for Multilingual Paraphrasing Using Adversarial Training
• Cross-Lingual Transfer Learning for Question Answering
• Evaluating the Fairness of Discriminative Foundation Models in Computer Vision
• On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models
• Enhanced Meta-Learning for Cross-lingual Named Entity Recognition with Minimal Resources
• Diverse Pretrained Context Encodings Improve Document Translation
• Incorporating Word and Subword Units in Unsupervised Machine Translation Using Language Model Rescoring
• Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation
• Complex QA and language models hybrid architectures, Survey
• Transformers to Learn Hierarchical Contexts in Multiparty Dialogue for Span-based Question Answering
• Robust Optimization for Multilingual Translation with Imbalanced Data
• MultiFiT: Efficient Multi-lingual Language Model Fine-tuning
• TADA: Task-Agnostic Dialect Adapters for English
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
• A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT
• Self-Supervised Dialogue Learning
• Enhancing Answer Boundary Detection for Multilingual Machine Reading Comprehension
• Proof Artifact Co-training for Theorem Proving with Language Models
• Beyond English-Only Reading Comprehension: Experiments in Zero-Shot Multilingual Transfer for Bulgarian
• Analysis of the Evolution of Advanced Transformer-Based Language Models: Experiments on Opinion Mining
• A Neighbourhood Framework for Resource-Lean Content Flagging
• Empowering News Recommendation with Pre-trained Language Models
• Outline to Story: Fine-grained Controllable Story Generation from Cascaded Events
• Offensive Language Identification in Low-resourced Code-mixed Dravidian languages using Pseudo-labeling
• Language Anisotropic Cross-Lingual Model Editing
• Bilingual Alignment Pre-Training for Zero-Shot Cross-Lingual Transfer
• Knowledge-Grounded Dialogue Generation with Pre-trained Language Models
• Style Attuned Pre-training and Parameter Efficient Fine-tuning for Spoken Language Understanding
• Can Monolingual Pretrained Models Help Cross-Lingual Classification?
• BitTensor: A Peer-to-Peer Intelligence Market
• Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER
• Denoising based Sequence-to-Sequence Pre-training for Text Generation
• Unsupervised Translation of German--Lower Sorbian: Exploring Training and Novel Transfer Methods on a Low-Resource Language
• Unsupervised Parallel Corpus Mining on Web Data
• Sinhala-English Parallel Word Dictionary Dataset
• Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural Networks
• SS-BERT: Mitigating Identity Terms Bias in Toxic Comment Classification by Utilising the Notion of "Subjectivity" and "Identity Terms"
• Machine Learning Meets Natural Language Processing -- The story so far
• IOT: Instance-wise Layer Reordering for Transformer Structures
• Revisiting Language Encoding in Learning Multilingual Representations
• ST-BERT: Cross-modal Language Model Pre-training For End-to-end Spoken Language Understanding
• Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation
• Transfer Learning for Multi-lingual Tasks -- a Survey
• UzBERT: pretraining a BERT model for Uzbek
• A Survey on Low-Resource Neural Machine Translation
• LAWDR: Language-Agnostic Weighted Document Representations from Pre-trained Models
• Is In-hospital Meta-information Useful for Abstractive Discharge Summary Generation?
• Backretrieval: An Image-Pivoted Evaluation Metric for Cross-Lingual Text Representations Without Parallel Corpora
• Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads
• Unsupervised Pre-training for Natural Language Generation: A Literature Review
• Cognate Transformer for Automated Phonological Reconstruction and Cognate Reflex Prediction
• Paraphrase Generation as Unsupervised Machine Translation
• DAMP: Doubly Aligned Multilingual Parser for Task-Oriented Dialogue
• A Few Topical Tweets are Enough for Effective User-Level Stance Detection
• EmotionGIF-Yankee: A Sentiment Classifier with Robust Model Based Ensemble Methods
• Fabricator: An Open Source Toolkit for Generating Labeled Training Data with Teacher LLMs
• Cross-lingual neural fuzzy matching for exploiting target-language monolingual corpora in computer-aided translation
• Can You Fool AI by Doing a 180? $\unicode{x2013}$ A Case Study on Authorship Analysis of Texts by Arata Osada
• Isomorphic Cross-lingual Embeddings for Low-Resource Languages
• Exploring the Promises of Transformer-Based LMs for the Representation of Normative Claims in the Legal Domain
• JASS: Japanese-specific Sequence to Sequence Pre-training for Neural Machine Translation
• A Survey of Syntactic-Semantic Parsing Based on Constituent and Dependency Structures
• El Departamento de Nosotros: How Machine Translated Corpora Affects Language Models in MRC Tasks
• Building Low-Resource NER Models Using Non-Speaker Annotation
• Long-Short Term Masking Transformer: A Simple but Effective Baseline for Document-level Neural Machine Translation
• Utilizing Resource-Rich Language Datasets for End-to-End Scene Text Recognition in Resource-Poor Languages
• Mischief: A Simple Black-Box Attack Against Transformer Architectures
• Explicit Cross-lingual Pre-training for Unsupervised Machine Translation
• Unsupervised Domain Adaptation of a Pretrained Cross-Lingual Language Model
• Learning to Emphasize: Dataset and Shared Task Models for Selecting Emphasis in Presentation Slides
• Analyzing Zero-shot Cross-lingual Transfer in Supervised NLP Tasks
• Lexical Simplification with Pretrained Encoders
• Document Sub-structure in Neural Machine Translation
• YES SIR!Optimizing Semantic Space of Negatives with Self-Involvement Ranker
• A Conditional Generative Matching Model for Multi-lingual Reply Suggestion
• Examining the rhetorical capacities of neural language models
• Self-Learning for Zero Shot Neural Machine Translation
• Predicting the Performance of Multilingual NLP Models
• Empirical Analysis of Korean Public AI Hub Parallel Corpora and in-depth Analysis using LIWC
• Explaining Documents' Relevance to Search Queries
• Leveraging Adversarial Training in Self-Learning for Cross-Lingual Text Classification
• MDQE: A More Accurate Direct Pretraining for Machine Translation Quality Estimation
• Priberam Labs at the NTCIR-15 SHINRA2020-ML: Classification Task
• Learning to Copy for Automatic Post-Editing
• Simple and Effective Input Reformulations for Translation
• Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Data
• Light Coreference Resolution for Russian with Hierarchical Discourse Features
• TransAug: Translate as Augmentation for Sentence Embeddings
• Analysis on Image Set Visual Question Answering
• Improving speech recognition models with small samples for air traffic control systems
• A Simple and Effective Approach to Automatic Post-Editing with Transfer Learning
• The LIG system for the English-Czech Text Translation Task of IWSLT 2019
• IntelliCAT: Intelligent Machine Translation Post-Editing with Quality Estimation and Translation Suggestion
• Using Emotion Embeddings to Transfer Knowledge Between Emotions, Languages, and Annotation Formats
• Public Health Informatics: Proposing Causal Sequence of Death Using Neural Machine Translation
• MvSR-NAT: Multi-view Subset Regularization for Non-Autoregressive Machine Translation
• Contributions to Representation Learning with Graph Autoencoders and Applications to Music Recommendation
• Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates
• Making Table Understanding Work in Practice