6 papers
Dual-objective Language Models: Training Efficiency Without Overfitting
David Samuel, Lucas Georges Gabriel Charpentier
This paper combines autoregressive and masked-diffusion training objectives without any architectural modifications, resulting in flexible language models that outperform single-ob…
Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languages
David Samuel, Lilja Ãvrelid, Erik Velldal +1
We propose a post-training method for lower-resource languages that preserves the fluency of language models even when aligned by disfluent reward models. Preference optimization i…
NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark
Vladislav Mikhailov, Tita Enstad, David Samuel +4
This paper introduces NorEval, a new and comprehensive evaluation suite for large-scale standardized benchmarking of Norwegian generative language models (LMs). NorEval consists of…
An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)
Laurie Burchell, Ona de Gibert, Nikolay Arefyev +32
Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In th…
Multi-label Scandinavian Language Identification (SLIDE)
Mariia Fedorova, Jonas Sebulon Frydenberg, Victoria Handford +6
Identifying closely related languages at sentence level is difficult, in particular because it is often impossible to assign a sentence to a single language. In this paper, we focu…
GPT or BERT: why not both?
Lucas Georges Gabriel Charpentier, David Samuel
We present a simple way to merge masked language modeling with causal language modeling. This hybrid training objective results in a model that combines the strengths of both model…