What the [MASK]? Making Sense of Language-Specific BERT Models

Abstract

Recently, Natural Language Processing (NLP) has witnessed an impressive progress in many areas, due to the advent of novel, pretrained contextual representation models. In particular, Devlin et al. (2019) proposed a model, called BERT (Bidirectional Encoder Representations from Transformers), which enables researchers to obtain state-of-the art performance on numerous NLP tasks by fine-tuning the representations on their data set and task, without the need for developing and training highly-specific architectures. The authors also released multilingual BERT (mBERT), a model trained on a corpus of 104 languages, which can serve as a universal language model. This model obtained impressive results on a zero-shot cross-lingual natural inference task. Driven by the potential of BERT models, the NLP community has started to investigate and generate an abundant number of BERT models that are trained on a particular language, and tested on a specific data domain and task. This allows us to evaluate the true potential of mBERT as a universal language model, by comparing it to the performance of these more specific models. This paper presents the current state of the art in language-specific BERT models, providing an overall picture with respect to different dimensions (i.e. architectures, data domains, and tasks). Our aim is to provide an immediate and straightforward overview of the commonalities and differences between Language-Specific (language-specific) BERT models and mBERT. We also provide an interactive and constantly updated website that can be used to explore the information we have collected, at https://bertlang.unibocconi.it.

References in corpus (8)

• Distributed Representations of Words and Phrases and their Compositionality
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• CamemBERT: a Tasty French Language Model
• AraBERT: Transformer-based Model for Arabic Language Understanding
• Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language
• BERTje: A Dutch BERT Model
• Multilingual is not enough: BERT for Finnish
• Binary and Multitask Classification Model for Dutch Anaphora Resolution: Die/Dat Prediction

Cited by in corpus (18)

• CamemBERT: a Tasty French Language Model
• A review of sentiment analysis research in Arabic language
• What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological properties
• Latin BERT: A Contextual Language Model for Classical Philology
• WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models
• Contrastive Language-Image Pre-training for the Italian Language
• Adapting Monolingual Models: Data can be Scarce when Language Similarity is High
• German's Next Language Model
• EstBERT: A Pretrained Language-Specific BERT for Estonian
• Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size
• Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalan
• BERT Goes Shopping: Comparing Distributional Models for Product Representations
• Text-based classification of interviews for mental health -- juxtaposing the state of the art
• Query2Prod2Vec Grounded Word Embeddings for eCommerce
• MergeDistill: Merging Pre-trained Language Models using Distillation
• Monolingual and Cross-Lingual Acceptability Judgments with the Italian CoLA corpus
• belabBERT: a Dutch RoBERTa-based language model applied to psychiatric classification
• Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference