Establishing Baselines for Text Classification in Low-Resource Languages

Abstract

While transformer-based finetuning techniques have proven effective in tasks that involve low-resource, low-data environments, a lack of properly established baselines and benchmark datasets make it hard to compare different approaches that are aimed at tackling the low-resource setting. In this work, we provide three contributions. First, we introduce two previously unreleased datasets as benchmark datasets for text classification and low-resource multilabel text classification for the low-resource language Filipino. Second, we pretrain better BERT and DistilBERT models for use within the Filipino setting. Third, we introduce a simple degradation test that benchmarks a model's resistance to performance degradation as the number of training samples are reduced. We analyze our pretrained model's degradation speeds and look towards the use of this method for comparing models aimed at operating within the low-resource setting. We release all our models and datasets for the research community to use.

We release all our models, finetuning code, and data at https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Understanding Back-Translation at Scale
• ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
• Linguistic Knowledge as Memory for Recurrent Neural Networks
• Evaluating Language Model Finetuning Techniques for Low-resource Languages
• Localization of Fake News Detection via Multitask Transfer Learning
• Semantics-aware BERT for Language Understanding
• Positional Encoding to Control Output Sequence Length

Cited by in corpus (8)

• Low-resource Languages: A Review of Past Work and Future Challenges
• Improving Large-scale Language Models and Resources for Filipino
• Cracking Double-Blind Review: Authorship Attribution with Deep Learning
• Exploiting All Samples in Low-Resource Sentence Classification: Early Stopping and Initialization Parameters
• Pagsusuri ng RNN-based Transfer Learning Technique sa Low-Resource Language
• BERT Embeddings for Automatic Readability Assessment
• IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation
• Exploiting News Article Structure for Automatic Corpus Generation of Entailment Datasets