10 papers
Verbalizing Activation Differences with Natural Language Autoencoders
Andrea Blasi Núñez, Gianluca Barmina, Federico Torrielli +3
Activation verbalizers translate the internal representations of large language models into human-readable text. The recently proposed natural language autoencoders (NLAs) learn th…
Multilingual GSM-Symbolic: What determines capability transfer across languages?
Kenneth Enevoldsen, Riley Herchert, Sofie Mosegaard +22
We understand little about how capabilities acquired in one language carry over to another, or what governs this transfer: evaluations rely on incomparable, saturation-prone datase…
Selecting The Most Informative Tokens in Natural Language Autoencoders
Federico Torrielli, Gianluca Barmina, Andrea Blasi Núñez +4
Natural language autoencoders translate a language model's internal activations into readable explanations. Explaining every token position is costly. Which positions should an aud…
Mimir: A Sub-2B Hierarchical Reasoning Model Trained on Permissible Data Punches Above Its Weight Class
Peter Schneider-Kamp, Jacob Nielsen, Gianluca Barmina +4
Training capable language models typically relies on web-scale pretraining followed by multi-stage post-training, creating substantial computational and data-governance barriers fo…
BrainSurgery: Reproducible and Flexible Declarative Weight Manipulations for Model Editing and Upcycling
Gianluca Barmina, Annemette Broch Pirchert, Andrea Blasi Núñez +2
Checkpoint transformations such as model merging, precision casting, and low-rank rewriting are commonly implemented through Python code or separate task-specific tools. We present…
PsychoSafe: Eliciting Psychologically-Informed Refusals in Large Language Models
Gianluca Barmina, Federico Torrielli, Sven Harms +8
Large language models (LLMs) routinely face requests that should be refused, creating a trade-off between helpfulness and harm prevention. However, refusals themselves can be helpf…
LLMs Can Leak Training Data But Do They Want To? Evaluating Memorization Propensity in LLMs
Gianluca Barmina, Peter Schneider-Kamp, Lukas Galke Poech
Large language models can reproduce training data, but memorization evaluations mostly measure whether models can be made to do so, not whether they do so without adversarial promp…
SommBench: Assessing Sommelier Expertise of Language Models
William Brach, Tomas Bedej, Jacob Nielsen +10
With the rapid advances of large language models, it becomes increasingly important to systematically evaluate their multilingual and multicultural capabilities. Previous cultural…
DaLA: Danish Linguistic Acceptability Evaluation Guided by Real World Errors
Gianluca Barmina, Nathalie Carmen Hau Norman, Peter Schneider-Kamp +1
We present an enhanced benchmark for evaluating linguistic acceptability in Danish. We first analyze the most common errors found in written Danish. Based on this analysis, we intr…
Dynaword: From One-shot to Continuously Developed Datasets
Kenneth Enevoldsen, Kristian Nørgaard Jensen, Jan Kostkan +14
Large-scale datasets are foundational for research and development in natural language processing. However, current approaches face three key challenges: (1) reliance on ambiguousl…