Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks

Abstract

We present Unicoder, a universal language encoder that is insensitive to different languages. Given an arbitrary NLP task, a model can be trained with Unicoder using training data in one language and directly applied to inputs of the same task in other languages. Comparing to similar efforts such as Multilingual BERT and XLM, three new cross-lingual pre-training tasks are proposed, including cross-lingual word recovery, cross-lingual paraphrase classification and cross-lingual masked language model. These tasks help Unicoder learn the mappings among different languages from more perspectives. We also find that doing fine-tuning on multiple languages together can bring further improvement. Experiments are performed on two tasks: cross-lingual natural language inference (XNLI) and cross-lingual question answering (XQA), where XLM is our baseline. On XNLI, 1.8% averaged accuracy improvement (on 15 languages) is obtained. On XQA, which is a new cross-lingual dataset built by us, 5.5% averaged accuracy improvement (on French and German) is obtained.

arXiv commentAccepted to EMNLP2019; 10 pages, 2 figures

References in corpus (7)

• Exploiting Similarities among Languages for Machine Translation
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Massively Multilingual Word Embeddings
• Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
• Empower Sequence Labeling with Task-Aware Neural Language Model
• Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
• Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT

Cited by in corpus (47)

• On the Cross-lingual Transferability of Monolingual Representations
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• Translation Artifacts in Cross-lingual Transfer Learning
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
• Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information
• M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training
• What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological properties
• Emerging Cross-lingual Structure in Pretrained Language Models
• XGPT: Cross-modal Generative Pre-Training for Image Captioning
• Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learning
• Are All Languages Created Equal in Multilingual BERT?
• Specializing Multilingual Language Models: An Empirical Study
• Pre-Trained Models: Past, Present and Future
• ProTo: Program-Guided Transformer for Program-Guided Tasks
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
• CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP
• Explicit Alignment Objectives for Multilingual Bidirectional Encoders
• Mixed-Lingual Pre-training for Cross-lingual Summarization
• Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translation
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
• Enhancing Answer Boundary Detection for Multilingual Machine Reading Comprehension
• Multilingual BERT Post-Pretraining Alignment
• Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
• Bilingual Alignment Pre-Training for Zero-Shot Cross-Lingual Transfer
• Do Explicit Alignments Robustly Improve Multilingual Encoders?
• ST-BERT: Cross-modal Language Model Pre-training For End-to-end Spoken Language Understanding
• Revisiting Language Encoding in Learning Multilingual Representations
• Toward Cross-Lingual Definition Generation for Language Learners
• Cross-lingual Text Classification with Heterogeneous Graph Neural Network
• Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots
• ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
• CalibreNet: Calibration Networks for Multilingual Sequence Labeling
• Cross-lingual Spoken Language Understanding with Regularized Representation Alignment
• MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
• Why Overfitting Isn't Always Bad: Retrofitting Cross-Lingual Word Embeddings to Dictionaries
• X-METRA-ADA: Cross-lingual Meta-Transfer Learning Adaptation to Natural Language Understanding and Question Answering
• X2Parser: Cross-Lingual and Cross-Domain Framework for Task-Oriented Compositional Semantic Parsing
• Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation
• Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
• LV-BERT: Exploiting Layer Variety for BERT
• Predicting the Performance of Multilingual NLP Models
• Let Your Heart Speak in its Mother Tongue: Multilingual Captioning of Cardiac Signals
• Learning from Multiple Noisy Augmented Data Sets for Better Cross-Lingual Spoken Language Understanding
• Discovering Representation Sprachbund For Multilingual Pre-Training
• Improved Multilingual Language Model Pretraining for Social Media Text via Translation Pair Prediction