Pre-Training BERT on Arabic Tweets: Practical Considerations
arXiv:2102.10684
Abstract
Pretraining Bidirectional Encoder Representations from Transformers (BERT) for downstream NLP tasks is a non-trival task. We pretrained 5 BERT models that differ in the size of their training sets, mixture of formal and informal Arabic, and linguistic preprocessing. All are intended to support Arabic dialects and social media. The experiments highlight the centrality of data diversity and the efficacy of linguistically aware segmentation. They also highlight that more data or more training step do not necessitate better models. Our new models achieve new state-of-the-art results on several downstream tasks. The resulting models are released to the community under the name QARiB.
6 pages, 5 figures
References in corpus (8)
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- AraBERT: Transformer-based Model for Arabic Language Understanding
- BERTje: A Dutch BERT Model
- Multilingual is not enough: BERT for Finnish
- Arabic Offensive Language on Twitter: Analysis and Experiments
- Multi-Dialect Arabic BERT for Country-Level Dialect Identification
- Arabic Dialect Identification in the Wild
- AnchiBERT: A Pre-Trained Model for Ancient ChineseLanguage Understanding and Generation