4 papers
A Sovereign, Open-Source Foundation Model for German and English
The Soofi-Team, Soofi-Team, : +31
We present Soofi S 30B-A3B, a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English. Its hybrid design activates only 3B…
The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning
Justinas Zaliaduonis, Patrick Putzky, Till Richter +1
Contrastive learning has become a leading paradigm for self-supervised representation learning, yet the conditions under which it recovers meaningful latent geometry remain incompl…
Float8@2bits: Entropy Coding Enables Data-Free Model Compression
Patrick Putzky, Martin Genzel, Mattes Mollenhauer +3
Post-training compression is currently divided into two contrasting regimes. On the one hand, fast, data-free, and model-agnostic methods (e.g., NF4 or HQQ) offer maximum accessibi…
Choose Your Model Size: Any Compression of Large Language Models Without Re-Computation
Martin Genzel, Patrick Putzky, Pengfei Zhao +5
The adoption of Foundation Models in resource-constrained environments remains challenging due to their large size and inference costs. A promising way to overcome these limitation…