1 citations · 2 across the 6 of their papers we have counts for
7 papers
LoopMTP: A looped transformer guided by latent multi-token prediction
Behzad Shomali, Markus Frey, David Berghaus +2
Looped transformers have emerged as a parameter-efficient alternative to scaling depth for strong reasoning. By reusing one stack of layers across iterations, they attain the e…
A Sovereign, Open-Source Foundation Model for German and English
Soofi-Team, :, Benedikt Droste +30
We present Soofi S 30B-A3B, a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English. Its hybrid design activates only 3B…
LLM Parameters for Math Across Languages: Shared or Separate?
Behzad Shomali, Luisa Victor, Tim Selbach +5
Large language models (LLMs) exhibit substantial cross-lingual variation in mathematical reasoning performance, but it remains unclear whether these differences reflect language-sp…
A Dual-Path Architecture for Scaling Compute and Capacity in LLMs
Markus Frey, Behzad Shomali, Joachim Koehler +1
Looped transformers apply a shared block multiple times and have emerged as a parameter-efficient route to scaling compute in language models. However, at fixed FLOPs a looped mode…
Adaptive Loops and Memory in Transformers: Think Harder or Know More?
Markus Frey, Behzad Shomali, Ali Hamza Bashir +3
Chain-of-thought (CoT) prompting enables reasoning in language models but requires explicit verbalization of intermediate steps. Looped transformers offer an alternative by iterati…
Is continuous CoT better suited for multi-lingual reasoning?
Ali Hamza Bashir, Behzad Shomali, Markus Frey +3
We investigate whether performing reasoning in a continuous latent space leads to more robust multilingual capabilities. We compare Continuous Chain-of-Thought (using the CODI fram…
Time-series attribution maps with regularized contrastive learning
Steffen Schneider, Rodrigo González Laiz, Anastasiia Filippova +2
Gradient-based attribution methods aim to explain decisions of deep learning models but so far lack identifiability guarantees. Here, we propose a method to generate attribution ma…