4 citations · 10 across the 10 of their papers we have counts for
3 papers · 1 filter
KletterMix: Climbing Toward High-Quality German Pretraining Data - The Full Report
Maurice Kraus, Ruben Härle, Sebastian Sztwiertnia +5
High-quality pretraining data is a central ingredient in modern language models, but German-language resources remain far less developed than their English counterparts: they are o…
A Sovereign, Open-Source Foundation Model for German and English
Soofi-Team, :, Benedikt Droste +30
We present Soofi S 30B-A3B, a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English. Its hybrid design activates only 3B…
Modalities, a PyTorch-native Framework For Large-scale LLM Training and Research
Max Lübbering, Timm Ruland, Richard Rutmann +8
Today's LLM (pre-) training and research workflows typically allocate a significant amount of compute to large-scale ablation studies. Despite the substantial compute costs of thes…