6 citations · 7 across the 4 of their papers we have counts for
4 papers
An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)
Laurie Burchell, Ona de Gibert, Nikolay Arefyev +32
Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In th…
FastSpell: the LangId Magic Spell
Marta Bañón, Jaume Zaragoza-Bernabeu, Gema Ramírez-Sánchez +1
Language identification is a crucial component in the automated production of language resources, particularly in multilingual and big data contexts. However, commonly used languag…
A New Massive Multilingual Dataset for High-Performance Language Technologies
Ona de Gibert, Graeme Nail, Nikolay Arefyev +10
We present the HPLT (High Performance Language Technologies) language resources, a new massive multilingual dataset including both monolingual and bilingual corpora extracted from…
OpusCleaner and OpusTrainer, open source toolkits for training Machine Translation and Large language models
Nikolay Bogoychev, Jelmer van der Linde, Graeme Nail +7
Developing high quality machine translation systems is a labour intensive, challenging and confusing process for newcomers to the field. We present a pair of tools OpusCleaner and…