5 papers
The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP
Henry He, Johann Frei, Raphael Schmitt
Digital healthcare generates vast amounts of clinical text that can support AI-assisted applications, yet German biomedical language models remain limited by older architectures or…
PortBERT: Navigating the Depths of Portuguese Language Models
Raphael Scheible-Schmitt, Henry He, Armando B. Mendes
Transformer models dominate modern NLP, but efficient, language-specific models remain scarce. In Portuguese, most focus on scale or accuracy, often neglecting training and deploym…
HalleluBERT: Let Every Token That Has Meaning Bear Its Weight
Raphael Schmitt
Transformer-based models have advanced NLP, yet Hebrew still lacks a RoBERTa encoder that is trained at scale and released in both base and large variants. We present HalleluBERT,…
SindBERT, the Sailor: Charting the Seas of Turkish NLP
Raphael Schmitt, Stefan Schweter
Transformer models have revolutionized NLP, yet many morphologically rich languages remain underrepresented in large-scale pre-training efforts. With SindBERT, we set out to chart…
GeistBERT: Breathing Life into German NLP
Raphael Scheible-Schmitt, Johann Frei
Advances in transformer-based language models have highlighted the benefits of language-specific pre-training on high-quality corpora. In this context, German NLP stands to gain fr…