16 papers · 1 filter
Verbalizing Activation Differences with Natural Language Autoencoders
Andrea Blasi Núñez, Gianluca Barmina, Federico Torrielli +3
Activation verbalizers translate the internal representations of large language models into human-readable text. The recently proposed natural language autoencoders (NLAs) learn th…
Selecting The Most Informative Tokens in Natural Language Autoencoders
Federico Torrielli, Gianluca Barmina, Andrea Blasi Núñez +4
Natural language autoencoders translate a language model's internal activations into readable explanations. Explaining every token position is costly. Which positions should an aud…
Stale-Document Poisoning: When Outdated Retrieval Overrides Correct Model Answers
Md Shamim Ahmed, Lukas Galke Poech, Richard Röttger
Retrieval-augmented generation (RAG) is often used to address outdated knowledge by providing external evidence. But retrieval helps only when that evidence is still valid. We iden…
Rethinking Pivot Programming Languages in Code Language Models
Andor Diera, Lukas Galke Poech, Matthias Tichy
Multilingual code language models transfer skills across programming languages (PLs), but whether any PL occupies a privileged pivot position remains contested: geometric analyses…
Mimir: A Sub-2B Hierarchical Reasoning Model Trained on Permissible Data Punches Above Its Weight Class
Peter Schneider-Kamp, Jacob Nielsen, Gianluca Barmina +4
Training capable language models typically relies on web-scale pretraining followed by multi-stage post-training, creating substantial computational and data-governance barriers fo…
PsychoSafe: Eliciting Psychologically-Informed Refusals in Large Language Models
Gianluca Barmina, Federico Torrielli, Sven Harms +7
Large language models (LLMs) routinely face requests that should be refused, creating a trade-off between helpfulness and harm prevention. However, refusals themselves can be helpf…
LLMs Can Leak Training Data But Do They Want To? Evaluating Memorization Propensity in LLMs
Gianluca Barmina, Peter Schneider-Kamp, Lukas Galke Poech
Large language models can reproduce training data, but memorization evaluations mostly measure whether models can be made to do so, not whether they do so without adversarial promp…
Emergent Languages in Populations of Language Model Agents: From Token Efficiency to Oversight Evasion
Stine Lyngsø Beltoft, William Brach, Federico Torrielli +5
Monitoring autonomous language model agents currently relies mostly on surface behavior. But what happens when agent populations invent new languages with the goal of avoiding huma…
ChronoMedKG: A Temporally-Grounded Biomedical Knowledge Graph and Benchmark for Clinical Reasoning
Md Shamim Ahmed, Farzaneh Firoozbakht, Lukas Galke Poech +2
Biomedical knowledge graphs (KGs) treat disease associations as static facts, but temporal information is crucial for clinical reasoning, e.g., a symptom diagnostic of one disease…
The Moltbook Files: A Harmless Slopocalypse or Humanity's Last Experiment
William Brach, Federico Torrielli, Stine Lyngsø Beltoft +3
Moltbook is a Reddit-like platform where OpenClaw agents post, comment, and vote at scale - a so far unprecedented incident that comes with serious safety concerns. With the aim of…
The Provenance Gap in Clinical AI: Evidence-Traceable Temporal Knowledge Graphs for Rare Disease Reasoning
Md Shamim Ahmed, Maja Dusanic, Moritz Nikolai Kirschner +4
Frontier large language models generate clinically accurate outputs, but their citations are often fabricated. We term this the Provenance Gap. We tested five frontier LLMs across…
Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals
Federico Torrielli, Peter Schneider-Kamp, Lukas Galke Poech
An activation oracle is a language model trained to convert the activations of another model into natural language. However, it is so far unclear to what extent these verbalization…
SommBench: Assessing Sommelier Expertise of Language Models
William Brach, Tomas Bedej, Jacob Nielsen +10
With the rapid advances of large language models, it becomes increasingly important to systematically evaluate their multilingual and multicultural capabilities. Previous cultural…
SDUs DAISY: A Benchmark for Danish Culture
Jacob Nielsen, Stine L. Beltoft, Peter Schneider-Kamp +1
We introduce Daisy, a factual knowledge benchmark for Danish cultural heritage, based on curated topics from the Danish Culture Canon 2006. For each artifact in the culture canon,…
DaLA: Danish Linguistic Acceptability Evaluation Guided by Real World Errors
Gianluca Barmina, Nathalie Carmen Hau Norman, Peter Schneider-Kamp +1
We present an enhanced benchmark for evaluating linguistic acceptability in Danish. We first analyze the most common errors found in written Danish. Based on this analysis, we intr…
Training Language Models to Use Prolog as a Tool
Niklas Mellgren, Peter Schneider-Kamp, Lukas Galke Poech
Language models frequently produce plausible yet incorrect reasoning traces that are difficult to verify. We investigate fine-tuning models to use Prolog as an external symbolic re…