1 citations · 2 across the 4 of their papers we have counts for
4 papers · 1 filter
Register Always Matters: Analysis of LLM Pretraining Data Through the Lens of Language Variation
Amanda Myntti, Erik Henriksson, Veronika Laippala +1
Pretraining data curation is a cornerstone in Large Language Model (LLM) development, leading to growing research on quality filtering of large web corpora. From statistical qualit…
An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)
Laurie Burchell, Ona de Gibert, Nikolay Arefyev +32
Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In th…
FinerWeb-10BT: Refining Web Data with LLM-Based Line-Level Filtering
Erik Henriksson, Otto Tarkka, Filip Ginter
Data quality is crucial for training Large Language Models (LLMs). Traditional heuristic filters often miss low-quality text or mistakenly remove valuable content. In this paper, w…
Automatic register identification for the open web using multilingual deep learning
Erik Henriksson, Amanda Myntti, Saara Hellström +3
This article presents multilingual deep learning models for identifying web registers -- text varieties such as news reports and discussion forums -- across 16 languages. We introd…