16 citations · 40 across the 17 of their papers we have counts for
13 papers · 1 filter
Kolibri: A Sovereign European Model on the Pareto Frontier
Aleph Alpha, :, Moritz Althaus +65
This report introduces Kolibri, an English-German mixture-of-experts transformer with 78.1B total parameters and 3.46B active parameters per token, released as open weights under t…
Rewarding Efficient Reasoning Improves Abstention on Underspecified Tasks in Reasoning Models
Polina Tsvilodub, Max Höth, Michael Franke +2
While modern large reasoning models (LRMs) excel at providing correct answers in many tasks, we provide additional evidence that they often struggle with a critical capability: kno…
Beyond Prompt and Parser Engineering: Soft-Prompt Tuning for Automated LLM Evaluation
Selen Erkan, Bastian Boll, Kristian Kersting +2
Benchmark scores often misrepresent a large language model's (LLM's) knowledge, because they rely, e.g., on the model's ability to follow specific formatting requirements. This esp…
AtManRL: Towards Faithful Reasoning via Differentiable Attention Saliency
Max Henning Höth, Kristian Kersting, Björn Deiseroth +1
Large language models (LLMs) increasingly rely on chain-of-thought (CoT) reasoning to solve complex tasks, yet a reasoning trace may merely accompany the final answer rather than c…
LIME: Making LLM Data More Efficient with Linguistic Metadata Embeddings
Sebastian Sztwiertnia, Felix Friedrich, Kristian Kersting +2
Pre-training decoder-only language models relies on vast amounts of high-quality data, yet the availability of such data is increasingly reaching its limits. While metadata is comm…
Bounding Hallucinations: Merlin-Arthur Protocols for Mutual-Information Bounds in Language Models
Björn Deiseroth, Max Henning Höth, Kristian Kersting +1
Retrieval-augmented generation (RAG) relies on retrieved context to guide large language models (LLM), yet treats the retrieval as a heuristic rather than verifiable evidence -- le…
Measuring and Guiding Monosemanticity
Ruben Härle, Felix Friedrich, Manuel Brack +4
There is growing interest in leveraging mechanistic interpretability and controllability to better understand and influence the internal dynamics of large language models (LLMs). H…
Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation
Thomas F Burns, Letitia Parcalabescu, Stephan Wäldchen +5
Scaling data quantity is essential for large language models (LLMs), yet recent findings show that data quality can significantly boost performance and training efficiency. We intr…
Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models
Pit Neitemeier, Björn Deiseroth, Constantin Eichenberg +1
Tokenization is a fundamental step in natural language processing, breaking text into units that computational models can process. While learned subword tokenizers have become the…
SCAR: Sparse Conditioned Autoencoders for Concept Detection and Steering in LLMs
Ruben Härle, Felix Friedrich, Manuel Brack +3
Large Language Models (LLMs) have demonstrated remarkable capabilities in generating human-like text, but their output may not be aligned with the user or even produce harmful cont…
T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddings
Björn Deiseroth, Manuel Brack, Patrick Schramowski +2
Tokenizers are crucial for encoding information in Large Language Models, but their development has recently stagnated, and they contain inherent weaknesses. Major limitations incl…
Divergent Token Metrics: Measuring degradation to prune away LLM components -- and optimize quantization
Björn Deiseroth, Max Meuer, Nikolas Gritsch +4
Large Language Models (LLMs) have reshaped natural language processing with their impressive capabilities. However, their ever-increasing size has raised concerns about their effec…
Do Multilingual Language Models Capture Differing Moral Norms?
Katharina Hämmerl, Björn Deiseroth, Patrick Schramowski +3
Massively multilingual sentence representations are trained on large corpora of uncurated data, with a very imbalanced proportion of languages included in the training. This may ca…