BERTje: A Dutch BERT Model

Abstract

The transformer-based pre-trained language model BERT has helped to improve state-of-the-art performance on many natural language processing (NLP) tasks. Using the same architecture and parameters, we developed and evaluated a monolingual Dutch BERT model called BERTje. Compared to the multilingual BERT model, which includes Dutch but is only based on Wikipedia text, BERTje is based on a large and diverse dataset of 2.4 billion tokens. BERTje consistently outperforms the equally-sized multilingual BERT model on downstream NLP tasks (part-of-speech tagging, named-entity recognition, semantic role labeling, and sentiment analysis). Our pre-trained Dutch BERT model is made available at https://github.com/wietsedv/bertje.

References in corpus (2)

• Multilingual is not enough: BERT for Finnish
• The merits of Universal Language Model Fine-tuning for Small Datasets -- a case with Dutch book reviews

Cited by in corpus (5)

• WikiBERT models: deep transfer learning for many languages
• The ADAPT Enhanced Dependency Parser at the IWPT 2020 Shared Task
• A Comparison of Supervised Learning to Match Methods for Product Search
• Transferring Monolingual Model to Low-Resource Language: The Case of Tigrinya
• Gender prediction using limited Twitter Data