4 papers
PortBERT: Navigating the Depths of Portuguese Language Models
Raphael Scheible-Schmitt, Henry He, Armando B. Mendes
Transformer models dominate modern NLP, but efficient, language-specific models remain scarce. In Portuguese, most focus on scale or accuracy, often neglecting training and deploym…
HalleluBERT: Let Every Token That Has Meaning Bear Its Weight
Raphael Schmitt
Transformer-based models have advanced NLP, yet Hebrew still lacks a RoBERTa encoder that is trained at scale and released in both base and large variants. We present HalleluBERT,…
SindBERT, the Sailor: Charting the Seas of Turkish NLP
Raphael Schmitt, Stefan Schweter
Transformer models have revolutionized NLP, yet many morphologically rich languages remain underrepresented in large-scale pre-training efforts. With SindBERT, we set out to chart…
GeistBERT: Breathing Life into German NLP
Raphael Scheible-Schmitt, Johann Frei
Advances in transformer-based language models have highlighted the benefits of language-specific pre-training on high-quality corpora. In this context, German NLP stands to gain fr…