Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages

Abstract

Multilingual transformer models like mBERT and XLM-RoBERTa have obtained great improvements for many NLP tasks on a variety of languages. However, recent works also showed that results from high-resource languages could not be easily transferred to realistic, low-resource scenarios. In this work, we study trends in performance for different amounts of available resources for the three African languages Hausa, isiXhosa and Yorùbá on both NER and topic classification. We show that in combination with transfer learning or distant supervision, these models can achieve with as little as 10 or 100 labeled sentences the same performance as baselines with much more supervised training data. However, we also find settings where this does not hold. Our discussions and additional experiments on assumptions such as time and hardware restrictions highlight challenges and opportunities in low-resource learning.

Accepted at EMNLP'20

References in corpus (11)

• Character-level Convolutional Networks for Text Classification
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition
• Snorkel: Rapid Training Data Creation with Weak Supervision
• Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• Learning with Noise: Enhance Distantly Supervised Relation Extraction with Dynamic Transition Matrix
• From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers
• Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set
• Weakly Supervised POS Taggers Perform Poorly on Truly Low-Resource Languages
• Matrix Smoothing: A Regularization for DNN with Transition Matrix under Noisy Labels

Cited by in corpus (8)

• Deep Learning Transformer Architecture for Named Entity Recognition on Low Resourced Languages: State of the art results
• XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages
• The first large scale collection of diverse Hausa language datasets
• ANEA: Distant Supervision for Low-Resource Named Entity Recognition
• ULF: Unsupervised Labeling Function Correction using Cross-Validation for Weak Supervision
• Learning with Noisy Labels by Adaptive Gradient-Based Outlier Removal
• Label-Descriptive Patterns and Their Application to Characterizing Classification Errors
• A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters