Pretraining Finnish ModernBERTs

Akseli Reunamo, Laura-Maria Peltonen, Hans Moen +1

This paper reports on pretraining ModernBERT encoder models in six different sizes, ranging from 51M to 475M parameters, with a focus on limited multilingualism, emphasizing langua…