BERTje: A Dutch BERT Model

Abstract

The transformer-based pre-trained language model BERT has helped to improve state-of-the-art performance on many natural language processing (NLP) tasks. Using the same architecture and parameters, we developed and evaluated a monolingual Dutch BERT model called BERTje. Compared to the multilingual BERT model, which includes Dutch but is only based on Wikipedia text, BERTje is based on a large and diverse dataset of 2.4 billion tokens. BERTje consistently outperforms the equally-sized multilingual BERT model on downstream NLP tasks (part-of-speech tagging, named-entity recognition, semantic role labeling, and sentiment analysis). Our pre-trained Dutch BERT model is made available at https://github.com/wietsedv/bertje.

References in corpus (2)

• Multilingual is not enough: BERT for Finnish
• The merits of Universal Language Model Fine-tuning for Small Datasets -- a case with Dutch book reviews

Cited by in corpus (51)

• Pre-trained Models for Natural Language Processing: A Survey
• AraBERT: Transformer-based Model for Arabic Language Understanding
• mT5: A massively multilingual pre-trained text-to-text transformer
• A Benchmark Study of Machine Learning Models for Online Fake News Detection
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• What the [MASK]? Making Sense of Language-Specific BERT Models
• Pre-Training BERT on Arabic Tweets: Practical Considerations
• FlauBERT: Unsupervised Language Model Pre-training for French
• What's so special about BERT's layers? A closer look at the NLP pipeline in monolingual and multilingual models
• LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• WikiBERT models: deep transfer learning for many languages
• Constructing Taxonomies from Pretrained Language Models
• Large-Scale Contextualised Language Modelling for Norwegian
• Adapting Monolingual Models: Data can be Scarce when Language Similarity is High
• FLERT: Document-Level Features for Named Entity Recognition
• Neural Proof Nets
• Evaluating Bias In Dutch Word Embeddings
• PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data
• A Benchmark of Rule-Based and Neural Coreference Resolution in Dutch Novels and News
• SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
• Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese
• Word differences in news media of lower and higher peace countries revealed by natural language processing and machine learning
• Czert -- Czech BERT-like Model for Language Representation
• Making sense of violence risk predictions using clinical notes
• Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languages
• X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
• The ADAPT Enhanced Dependency Parser at the IWPT 2020 Shared Task
• Multitask Learning for Low Resource Spoken Language Understanding
• Are the Multilingual Models Better? Improving Czech Sentiment with Transformers
• EstBERT: A Pretrained Language-Specific BERT for Estonian
• Combating the Curse of Multilinguality in Cross-Lingual WSD by Aligning Sparse Contextualized Word Representations
• Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages
• A Comparison of Supervised Learning to Match Methods for Product Search
• Dutch Named Entity Recognition and De-identification Methods for the Human Resource Domain
• Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalan
• Transferring Monolingual Model to Low-Resource Language: The Case of Tigrinya
• AnchiBERT: A Pre-Trained Model for Ancient ChineseLanguage Understanding and Generation
• Bilingual Language Modeling, A transfer learning technique for Roman Urdu
• Fighting the COVID-19 Infodemic: Modeling the Perspective of Journalists, Fact-Checkers, Social Media Platforms, Policy Makers, and the Society
• Real-time Streaming Wave-U-Net with Temporal Convolutions for Multichannel Speech Enhancement
• Improving BERT Pretraining with Syntactic Supervision
• ConveRT for FAQ Answering
• PyEuroVoc: A Tool for Multilingual Legal Document Classification with EuroVoc Descriptors
• KoreALBERT: Pretraining a Lite BERT Model for Korean Language Understanding
• Towards Fully Bilingual Deep Language Modeling
• A Logic-Based Framework for Natural Language Inference in Dutch
• belabBERT: a Dutch RoBERTa-based language model applied to psychiatric classification
• TMR: Evaluating NER Recall on Tough Mentions
• Predicting the Performance of Multilingual NLP Models
• Gender prediction using limited Twitter Data