RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model

Abstract

We present RobeCzech, a monolingual RoBERTa language representation model trained on Czech data. RoBERTa is a robustly optimized Transformer-based pretraining approach. We show that RobeCzech considerably outperforms equally-sized multilingual and Czech-trained contextualized language representation models, surpasses current state of the art in all five evaluated NLP tasks and reaches state-of-the-art results in four of them. The RobeCzech model is released publicly at https://hdl.handle.net/11234/1-3691 and https://huggingface.co/ufal/robeczech-base.

arXiv commentPublished in TSD 2021

References in corpus (6)

• Distributed Representations of Words and Phrases and their Compositionality
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• CamemBERT: a Tasty French Language Model
• Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation
• Multilingual is not enough: BERT for Finnish
• Czert -- Czech BERT-like Model for Language Representation

Cited by in corpus (5)

• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• Mono vs Multilingual BERT for Hate Speech Detection and Text Classification: A Case Study in Marathi
• CsFEVER and CTKFacts: Acquiring Czech data for fact verification
• Unsupervised extraction, labelling and clustering of segments from clinical notes
• Comparison of Czech Transformers on Text Classification Tasks