Evaluating Language Model Finetuning Techniques for Low-resource Languages

Abstract

Unlike mainstream languages (such as English and French), low-resource languages often suffer from a lack of expert-annotated corpora and benchmark resources that make it hard to apply state-of-the-art techniques directly. In this paper, we alleviate this scarcity problem for the low-resourced Filipino language in two ways. First, we introduce a new benchmark language modeling dataset in Filipino which we call WikiText-TL-39. Second, we show that language model finetuning techniques such as BERT and ULMFiT can be used to consistently train robust classifiers in low-resource settings, experiencing at most a 0.0782 increase in validation error when the number of training examples is decreased from 10K to 1K while finetuning using a privately-held sentiment dataset.

Pretrained models and datasets available at https://github.com/jcblaisecruz02/Tagalog-BERT

References in corpus (3)

• Regularizing and Optimizing LSTM Language Models
• The Natural Language Decathlon: Multitask Learning as Question Answering
• Low-Resource Named Entity Recognition with Cross-Lingual, Character-Level Neural Conditional Random Fields

Cited by in corpus (7)

• Establishing Baselines for Text Classification in Low-Resource Languages
• Sharing to learn and learning to share; Fitting together Meta-Learning, Multi-Task Learning, and Transfer Learning: A meta review
• Sentiment Classification in Swahili Language Using Multilingual BERT
• Improving Large-scale Language Models and Resources for Filipino
• Pagsusuri ng RNN-based Transfer Learning Technique sa Low-Resource Language
• BERT Embeddings for Automatic Readability Assessment
• Exploiting News Article Structure for Automatic Corpus Generation of Entailment Datasets