AraBERT: Transformer-based Model for Arabic Language Understanding

Abstract

The Arabic language is a morphologically rich language with relatively few resources and a less explored syntax compared to English. Given these limitations, Arabic Natural Language Processing (NLP) tasks like Sentiment Analysis (SA), Named Entity Recognition (NER), and Question Answering (QA), have proven to be very challenging to tackle. Recently, with the surge of transformers based models, language-specific BERT based models have proven to be very efficient at language understanding, provided they are pre-trained on a very large corpus. Such models were able to set new standards and achieve state-of-the-art results for most NLP tasks. In this paper, we pre-trained BERT specifically for the Arabic language in the pursuit of achieving the same success that BERT did for the English language. The performance of AraBERT is compared to multilingual BERT from Google and other state-of-the-art approaches. The results showed that the newly developed AraBERT achieved state-of-the-art performance on most tested Arabic NLP tasks. The pretrained araBERT models are publicly available on https://github.com/aub-mind/arabert hoping to encourage research and applications for Arabic NLP.

Proceedings of the Twelfth International Conference on Language Resources and Evaluation (LREC 2020), Marseille, France (2020)

References in corpus (7)

• Distributed Representations of Words and Phrases and their Compositionality
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• CamemBERT: a Tasty French Language Model
• Towards a Human-like Open-Domain Chatbot
• BERTje: A Dutch BERT Model
• 1.5 billion words Arabic Corpus
• ArSentD-LEV: A Multi-Topic Corpus for Target-based Sentiment Analysis in Arabic Levantine Tweets

Cited by in corpus (46)

• Pre-trained Models for Natural Language Processing: A Survey
• A Benchmark Study of Machine Learning Models for Online Fake News Detection
• Transformers in Healthcare: A Survey
• The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models
• What the [MASK]? Making Sense of Language-Specific BERT Models
• Arabic Fake News Detection Based on Deep Contextualized Embedding Models
• Arabic Offensive Language on Twitter: Analysis and Experiments
• Pre-Training BERT on Arabic Tweets: Practical Considerations
• FlauBERT: Unsupervised Language Model Pre-training for French
• Machine Generation and Detection of Arabic Manipulated and Fake News
• Pre-trained Language Models in Biomedical Domain: A Systematic Survey
• HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition
• Empirical evaluation of shallow and deep learning classifiers for Arabic sentiment analysis
• ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoders
• WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models
• ArabGlossBERT: Fine-Tuning BERT on Context-Gloss Pairs for WSD
• A Survey of COVID-19 Misinformation: Datasets, Detection Techniques and Open Issues
• Reducing Traffic Wastage in Video Streaming via Bandwidth-Efficient Bitrate Adaptation
• ArCOV19-Rumors: Arabic COVID-19 Twitter Dataset for Misinformation Detection
• Spam and Sentiment Detection in Arabic Tweets Using MARBERT Model
• Language Modelling with Pixels
• AraCOVID19-MFH: Arabic COVID-19 Multi-label Fake News and Hate Speech Detection Dataset
• Accenture at CheckThat! 2020: If you say so: Post-hoc fact-checking of claims using transformer-based models
• Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese
• PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data
• A Novel Dialect-Aware Framework for the Classification of Arabic Dialects and Emotions
• DziriBERT: a Pre-trained Language Model for the Algerian Dialect
• Quantization-Aware and Tensor-Compressed Training of Transformers for Natural Language Understanding
• Arabic Dialect Identification Using BERT-Based Domain Adaptation
• ANER: Arabic and Arabizi Named Entity Recognition using Transformer-Based Approach
• Combining Context-Free and Contextualized Representations for Arabic Sarcasm Detection and Sentiment Identification
• Comparative Approaches to Sentiment Analysis Using Datasets in Major European and Arabic Languages
• Transfer Learning for Multi-lingual Tasks -- a Survey
• An Empirical Study of Pre-trained Transformers for Arabic Information Extraction
• Rosetta Stone at KSAA-RD Shared Task: A Hop From Language Modeling To Word--Definition Alignment
• AnchiBERT: A Pre-Trained Model for Ancient ChineseLanguage Understanding and Generation
• Anonymous Hyperlocal Communities: What do they talk about?
• Bilingual Language Modeling, A transfer learning technique for Roman Urdu
• DiaLex: A Benchmark for Evaluating Multidialectal Arabic Word Embeddings
• Neural Coreference Resolution for Arabic
• belabBERT: a Dutch RoBERTa-based language model applied to psychiatric classification
• Supporting Undotted Arabic with Pre-trained Language Models
• Multilingual and Multilabel Emotion Recognition using Virtual Adversarial Training
• Using Emotion Embeddings to Transfer Knowledge Between Emotions, Languages, and Annotation Formats
• Findings of the NLP4IF-2021 Shared Tasks on Fighting the COVID-19 Infodemic and Censorship Detection
• Adapting MARBERT for Improved Arabic Dialect Identification: Submission to the NADI 2021 Shared Task