5 papers
A Sovereign, Open-Source Foundation Model for German and English
The Soofi-Team, Soofi-Team, : +31
We present Soofi S 30B-A3B, a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English. Its hybrid design activates only 3B…
ReasonXL: Shifting LLM Reasoning Language Without Sacrificing Performance
Daniil Gurgurov, Tom Röhr, Sebastian von Rohrscheidt +3
Despite advances in multilingual capabilities, most large language models (LLMs) remain English-centric in their training and, crucially, in their production of reasoning traces. E…
Robust Weight Imprinting: Insights from Neural Collapse and Proxy-Based Aggregation
Justus Westerhoff, Golzar Atefi, Mario Koddenbrock +4
The capacity of foundation models allows for their application to new, unseen tasks. The adaptation to such tasks is called transfer learning. An efficient transfer learning method…
Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation
Bogdan KostiÄ, Conor Fallon, Julian Risch +1
The rapid advancement of Large Language Models (LLMs) has established standardized evaluation benchmarks as the primary instrument for model comparison. Yet, their reliability is i…
Comply: Learning Sentences with Complex Weights inspired by Fruit Fly Olfaction
Alexei Figueroa, Justus Westerhoff, Golzar Atefi +5
Biologically inspired neural networks offer alternative avenues to model data distributions. FlyVec is a recent example that draws inspiration from the fruit fly's olfactory circui…