5 papers
Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training
Jingwei Zuo, Cong Zeng, Ilyas Chahed +6
The training paradigm of large language models has shifted from traditional one-pass training to multi-epoch training, as reasonable reuse of limited high-quality data can improve…
Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers
Maksim Velikanov, Ilyas Chahed, Jingwei Zuo +3
Applying weight decay (WD) to matrix layers is standard practice in large-language-model pretraining. Prior work suggests that stochastic gradient noise induces a Brownian-like exp…
Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance
Jingwei Zuo, Maksim Velikanov, Ilyas Chahed +24
In this report, we introduce Falcon-H1, a new series of large language models (LLMs) featuring hybrid architecture designs optimized for both high performance and efficiency across…
NeurIPS 2025 E2LM Competition : Early Training Evaluation of Language Models
Mouadh Yagoubi, Yasser Dahou, Billel Mokeddem +12
Existing benchmarks have proven effective for assessing the performance of fully trained large language models. However, we find striking differences in the early training stages o…
Falcon Mamba: The First Competitive Attention-free 7B Language Model
Jingwei Zuo, Maksim Velikanov, Dhia Eddine Rhaiem +4
In this technical report, we present Falcon Mamba 7B, a new base large language model based on the novel Mamba architecture. Falcon Mamba 7B is trained on 5.8 trillion tokens with…