7 papers
DFM Mimir v1: An Open HRM Delivering Frontier Performance at 1B Parameters Using Only Permissible Post-Training Data
Peter Schneider-Kamp, Jacob Nielsen, Gianluca Barmina +2
Current large language model development relies on massive, often non-permissible datasets, creating a high barrier for researchers committed to open-source and ethically sourced d…
BrainSurgery: Reproducible and Reliable Declarative Weight Manipulations for Model Editing and Upcycling
Gianluca Barmina, Annemette Broch Pirchert, Andrea Blasi Núñez +2
As deep learning models scale, managing, inspecting, and modifying large checkpoints has become increasingly challenging. Researchers often need to alter model weights for layer re…
PsychoSafe: Eliciting Psychologically-Informed Refusals in Large Language Models
Gianluca Barmina, Federico Torrielli, Sven Harms +7
Large language models (LLMs) routinely face requests that should be refused, creating a trade-off between helpfulness and harm prevention. However, refusals themselves can be helpf…
LLMs Can Leak Training Data But Do They Want To? A Propensity-Aware Evaluation of Memorization in LLMs
Gianluca Barmina, Peter Schneider-Kamp, Lukas Galke Poech
Large language models can reproduce training data, but existing memorization evaluations mostly measure whether models can be forced to do so, rather than whether they do so under…
SommBench: Assessing Sommelier Expertise of Language Models
William Brach, Tomas Bedej, Jacob Nielsen +10
With the rapid advances of large language models, it becomes increasingly important to systematically evaluate their multilingual and multicultural capabilities. Previous cultural…
DaLA: Danish Linguistic Acceptability Evaluation Guided by Real World Errors
Gianluca Barmina, Nathalie Carmen Hau Norman, Peter Schneider-Kamp +1
We present an enhanced benchmark for evaluating linguistic acceptability in Danish. We first analyze the most common errors found in written Danish. Based on this analysis, we intr…