4 papers
Structure Retention in Embedding Spaces as a Predictor of Benchmark Performance
Amanda Myntti, Jenna Kanerva, Veronika Laippala +1
In this paper, we show that high-performing embedding models organize their embedding spaces in a consistent way. We evaluate 25 contemporary embedding models on five MTEB tasks sp…
HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models
Stephan Oepen, Nikolay Arefev, Mikko Aulamo +29
We present an ongoing initiative to provide open, very large, high-quality, and richly annotated textual datasets for almost 200 languages. At 30 trillion tokens, this is likely th…
Automatic register identification for the open web using multilingual deep learning
Erik Henriksson, Amanda Myntti, Saara Hellström +3
This article presents multilingual deep learning models for identifying web registers -- text varieties such as news reports and discussion forums -- across 16 languages. We introd…
Register Always Matters: Analysis of LLM Pretraining Data Through the Lens of Language Variation
Amanda Myntti, Erik Henriksson, Veronika Laippala +1
Pretraining data curation is a cornerstone in Large Language Model (LLM) development, leading to growing research on quality filtering of large web corpora. From statistical qualit…