Czert -- Czech BERT-like Model for Language Representation
arXiv:2103.13031
Abstract
This paper describes the training process of the first Czech monolingual language representation models based on BERT and ALBERT architectures. We pre-train our models on more than 340K of sentences, which is 50 times more than multilingual models that include Czech data. We outperform the multilingual models on 9 out of 11 datasets. In addition, we establish the new state-of-the-art results on nine datasets. At the end, we discuss properties of monolingual and multilingual models based upon our results. We publish all the pre-trained and fine-tuned models freely for the research community.
arXiv comment13 pages
References in corpus (10)
- Adam: A Method for Stochastic Optimization
- Distributed Representations of Words and Phrases and their Compositionality
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- CamemBERT: a Tasty French Language Model
- Horovod: fast and easy distributed deep learning in TensorFlow
- BERTje: A Dutch BERT Model
- Simple BERT Models for Relation Extraction and Semantic Role Labeling
- Multilingual is not enough: BERT for Finnish
- FlauBERT: Unsupervised Language Model Pre-training for French