4 papers · 1 filter
A Sovereign, Open-Source Foundation Model for German and English
The Soofi-Team, Soofi-Team, : +31
We present Soofi S 30B-A3B, a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English. Its hybrid design activates only 3B…
Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs
Mehdi Ali, Michael Fromm, Klaudia Thellmann +38
We present two multilingual LLMs, Teuken 7B-base and Teuken 7B-instruct, designed to embrace Europe's linguistic diversity by supporting all 24 official languages of the European U…
Data Processing for the OpenGPT-X Model Family
Nicolo' Brandizzi, Hammam Abdelwahab, Anirban Bhowmick +19
This paper presents a comprehensive overview of the data preparation pipeline developed for the OpenGPT-X project, a large-scale initiative aimed at creating open and high-performa…
Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models
Mehdi Ali, Manuel Brack, Max Lübbering +15
High-quality multilingual training data is essential for effectively pretraining large language models (LLMs). Yet, the availability of suitable open-source multilingual datasets r…