9 papers · 1 filter
news-crawler-LM: A Small Long-Context Model For High-Quality News Crawling
Pascal Stolzenburg, Jonas Golde, Max Dallabetta +1
Extracting structured content from news pages remains challenging due to heterogeneous HTML layouts, inconsistent markup, and substantial boilerplate such as navigation elements an…
What Matters in Linearizing Language Models? A Comparative Study of Architecture, Scale, and Task Adaptation
Patrick Haller, Jonas Golde, Alan Akbik
Linearization has emerged as a strategy for developing efficient language models (LMs). Starting from an existing Transformer-based LM, linearization replaces the attention compone…
What Matters When Building Universal Multilingual Named Entity Recognition Models?
Jonas Golde, Patrick Haller, Alan Akbik
Recent progress in universal multilingual named entity recognition (NER) has been driven by multilingual transformer models, task-specific architectures, custom loss functions, and…
FiNERweb: Datasets and Artifacts for Scalable Multilingual Named Entity Recognition
Jonas Golde, Patrick Haller, Alan Akbik
Recent multilingual named entity recognition (NER) work has shown that large language models (LLMs) can provide effective synthetic supervision, yet such datasets have mostly appea…
Sample-Efficient Language Modeling with Linear Attention and Lightweight Enhancements
Patrick Haller, Jonas Golde, Alan Akbik
We study architectural and optimization techniques for sample-efficient language modeling under the constraints of the BabyLM 2025 shared task. Our model, BLaLM, replaces self-atte…
Question Decomposition for Retrieval-Augmented Generation
Paul J. L. Ammann, Jonas Golde, Alan Akbik
Grounding large language models (LLMs) in verifiable external sources is a well-established strategy for generating reliable answers. Retrieval-augmented generation (RAG) is one su…