BERTić -- The Transformer Language Model for Bosnian, Croatian, Montenegrin and Serbian

Nikola Ljubešić, Davor Lauc

In this paper we describe a transformer model pre-trained on 8 billion tokens of crawled text from the Croatian, Bosnian, Serbian and Montenegrin web domains. We evaluate the trans…