Bilingual Language Modeling, A transfer learning technique for Roman Urdu

Abstract

Pretrained language models are now of widespread use in Natural Language Processing. Despite their success, applying them to Low Resource languages is still a huge challenge. Although Multilingual models hold great promise, applying them to specific low-resource languages e.g. Roman Urdu can be excessive. In this paper, we show how the code-switching property of languages may be used to perform cross-lingual transfer learning from a corresponding high resource language. We also show how this transfer learning technique termed Bilingual Language Modeling can be used to produce better performing models for Roman Urdu. To enable training and experimentation, we also present a collection of novel corpora for Roman Urdu extracted from various sources and social networking sites, e.g. Twitter. We train Monolingual, Multilingual, and Bilingual models of Roman Urdu - the proposed bilingual model achieves 23% accuracy compared to the 2% and 11% of the monolingual and multilingual models respectively in the Masked Language Modeling (MLM) task.

References in corpus (15)

• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• CamemBERT: a Tasty French Language Model
• AraBERT: Transformer-based Model for Arabic Language Understanding
• A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language
• BERTje: A Dutch BERT Model
• Know What You Don't Know: Unanswerable Questions for SQuAD
• Multilingual Models for Compositional Distributed Semantics
• XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation
• A Strong Baseline for Learning Cross-Lingual Word Embeddings from Sentence Alignments