4 citations · 14 across the 13 of their papers we have counts for
13 papers
A Sovereign, Open-Source Foundation Model for German and English
Soofi-Team, :, Benedikt Droste +30
We present Soofi S 30B-A3B, a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English. Its hybrid design activates only 3B…
Modalities, a PyTorch-native Framework For Large-scale LLM Training and Research
Max Lübbering, Timm Ruland, Richard Rutmann +8
Today's LLM (pre-) training and research workflows typically allocate a significant amount of compute to large-scale ablation studies. Despite the substantial compute costs of thes…
Towards End-to-End Model-Agnostic Explanations for RAG Systems
Viju Sudhi, Sinchana Ramakanth Bhat, Max Rudat +2
Retrieval Augmented Generation (RAG) systems, despite their growing popularity for enhancing model response reliability, often struggle with trustworthiness and explainability. In…
Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models
Mehdi Ali, Manuel Brack, Max Lübbering +15
High-quality multilingual training data is essential for effectively pretraining large language models (LLMs). Yet, the availability of suitable open-source multilingual datasets r…
Rethinking Chunk Size For Long-Document Retrieval: A Multi-Dataset Analysis
Sinchana Ramakanth Bhat, Max Rudat, Jannis Spiekermann +1
Chunking is a crucial preprocessing step in retrieval-augmented generation (RAG) systems, significantly impacting retrieval effectiveness across diverse datasets. In this study, we…
Towards Multilingual LLM Evaluation for European Languages
Klaudia Thellmann, Bernhard Stadler, Michael Fromm +8
The rise of Large Language Models (LLMs) has revolutionized natural language processing across numerous languages and tasks. However, evaluating LLM performance in a consistent and…
Data Processing for the OpenGPT-X Model Family
Nicolo' Brandizzi, Hammam Abdelwahab, Anirban Bhowmick +19
This paper presents a comprehensive overview of the data preparation pipeline developed for the OpenGPT-X project, a large-scale initiative aimed at creating open and high-performa…
Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs
Mehdi Ali, Michael Fromm, Klaudia Thellmann +38
We present two multilingual LLMs, Teuken 7B-base and Teuken 7B-instruct, designed to embrace Europe's linguistic diversity by supporting all 24 official languages of the European U…
Fact Finder -- Enhancing Domain Expertise of Large Language Models by Incorporating Knowledge Graphs
Daniel Steinigen, Roman Teucher, Timm Heine Ruland +6
Recent advancements in Large Language Models (LLMs) have showcased their proficiency in answering natural language queries. However, their effectiveness is hindered by limited doma…
Do Multilingual Large Language Models Mitigate Stereotype Bias?
Shangrui Nie, Michael Fromm, Charles Welch +7
While preliminary findings indicate that multilingual LLMs exhibit reduced bias compared to monolingual ones, a comprehensive understanding of the effect of multilingual training o…
Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions?
Alexander Arno Weber, Klaudia Thellmann, Jan Ebert +4
The adaption of multilingual pre-trained LLMs into eloquent and helpful assistants is essential to facilitate their use across different language regions. In that spirit, we are th…
Tokenizer Choice For LLM Training: Negligible or Crucial?
Mehdi Ali, Michael Fromm, Klaudia Thellmann +18
The recent success of Large Language Models (LLMs) has been predominantly driven by curating the training dataset composition, scaling of model architectures and dataset sizes and…
Integrating Generative Artificial Intelligence in Intelligent Vehicle Systems
Lukas Stappen, Jeremy Dillmann, Serena Striegel +3
This paper aims to serve as a comprehensive guide for researchers and practitioners, offering insights into the current state, potential applications, and future research direction…