Specializing Multilingual Language Models: An Empirical Study

Abstract

Pretrained multilingual language models have become a common tool in transferring NLP capabilities to low-resource languages, often with adaptations. In this work, we study the performance, extensibility, and interaction of two such adaptations: vocabulary augmentation and script transliteration. Our evaluations on part-of-speech tagging, universal dependency parsing, and named entity recognition in nine diverse low-resource languages uphold the viability of these approaches while raising new questions around how to optimally adapt multilingual models to low-resource settings.

Workshop on Multilingual Representation Learning (MRL) 2021

References in corpus (10)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Cross-lingual Language Model Pretraining
• On the Cross-lingual Transferability of Monolingual Representations
• mT5: A massively multilingual pre-trained text-to-text transformer
• Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection
• Cross-Lingual Ability of Multilingual BERT: An Empirical Study
• Multilingual Alignment of Contextual Word Representations
• CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
• Parsing with Multilingual BERT, a Small Corpus, and a Small Treebank
• Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfer

Cited by in corpus (3)

• Improving Word Translation via Two-Stage Contrastive Learning
• Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese
• FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models