Alan Akbik

news-crawler-LM: A Small Long-Context Model For High-Quality News Crawling

Pascal Stolzenburg, Jonas Golde, Max Dallabetta +1

Extracting structured content from news pages remains challenging due to heterogeneous HTML layouts, inconsistent markup, and substantial boilerplate such as navigation elements an…

Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling

Ansar Aynetdinov, Patrick Haller, Alan Akbik

Recent research has shown that filtering massive English web corpora into high-quality subsets significantly improves training efficiency. However, for high-resource non-English la…

Beyond Marginal Distributions: A Framework to Evaluate the Representativeness of Demographic-Aligned LLMs

Tristan Williams, Franziska Weeber, Sebastian Padó +1

Large language models are increasingly used to represent human opinions, values, or beliefs, and their steerability towards these ideals is an active area of research. Existing wor…

What Matters When Building Universal Multilingual Named Entity Recognition Models?

Jonas Golde, Patrick Haller, Alan Akbik

Recent progress in universal multilingual named entity recognition (NER) has been driven by multilingual transformer models, task-specific architectures, custom loss functions, and…

FiNERweb: Datasets and Artifacts for Scalable Multilingual Named Entity Recognition

Jonas Golde, Patrick Haller, Alan Akbik

Recent multilingual named entity recognition (NER) work has shown that large language models (LLMs) can provide effective synthetic supervision, yet such datasets have mostly appea…

Sample-Efficient Language Modeling with Linear Attention and Lightweight Enhancements

Patrick Haller, Jonas Golde, Alan Akbik

We study architectural and optimization techniques for sample-efficient language modeling under the constraints of the BabyLM 2025 shared task. Our model, BLaLM, replaces self-atte…

Towards a Principled Evaluation of Knowledge Editors

Sebastian Pohl, Max Ploner, Alan Akbik

Model editing has been gaining increasing attention over the past few years. For Knowledge Editing in particular, more challenging evaluation datasets have recently been released.…

cs.CL2025★ 1 cited

Question Decomposition for Retrieval-Augmented Generation

Paul J. L. Ammann, Jonas Golde, Alan Akbik

Grounding large language models (LLMs) in verifiable external sources is a well-established strategy for generating reliable answers. Retrieval-augmented generation (RAG) is one su…

From Data to Knowledge: Evaluating How Efficiently Language Models Learn Facts

Daniel Christoph, Max Ploner, Patrick Haller +1

Sample efficiency is a crucial property of language models with practical implications for training efficiency. In real-world text, information follows a long-tailed distribution.…

What Matters in Linearizing Language Models? A Comparative Study of Architecture, Scale, and Task Adaptation

Patrick Haller, Jonas Golde, Alan Akbik

Linearization has emerged as a strategy for developing efficient language models (LMs). Starting from an existing Transformer-based LM, linearization replaces the attention compone…

MastermindEval: A Simple But Scalable Reasoning Benchmark

Jonas Golde, Patrick Haller, Fabio Barth +1

Recent advancements in large language models (LLMs) have led to remarkable performance across a wide range of language understanding and mathematical tasks. As a result, increasing…

BabyHGRN: Exploring RNNs for Sample-Efficient Training of Language Models

Patrick Haller, Jonas Golde, Alan Akbik

This paper explores the potential of recurrent neural networks (RNNs) and other subquadratic architectures as competitive alternatives to transformer-based models in low-resource l…

Familiarity: Better Evaluation of Zero-Shot Named Entity Recognition by Quantifying Label Shifts in Synthetic Training Data

Jonas Golde, Patrick Haller, Max Ploner +3

Zero-shot named entity recognition (NER) is the task of detecting named entities of specific types (such as 'Person' or 'Medicine') without any training examples. Current research…

TransformerRanker: A Tool for Efficiently Finding the Best-Suited Language Models for Downstream Classification Tasks

Lukas Garbas, Max Ploner, Alan Akbik

Classification tasks in NLP are typically addressed by selecting a pre-trained language model (PLM) from a model hub, and fine-tuning it for the task at hand. However, given the ve…

LM-PUB-QUIZ: A Comprehensive Framework for Zero-Shot Evaluation of Relational Knowledge in Language Models

Max Ploner, Jacek Wiland, Sebastian Pohl +1

Knowledge probing evaluates the extent to which a language model (LM) has acquired relational knowledge during its pre-training phase. It provides a cost-effective means of compari…

Large-Scale Label Interpretation Learning for Few-Shot Named Entity Recognition

Jonas Golde, Felix Hamborg, Alan Akbik

Few-shot named entity recognition (NER) detects named entities within text using only a few annotated examples. One promising line of research is to leverage natural language descr…