3 papers
cs.CL2026
Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model
Mohammed Sabry, Sean Augenstein, Keith Rush +1
We ask whether language-model pre-training can be decomposed into smaller, independently trainable jobs that can later be recomposed into a coherent larger model. We introduce Mixt…
cs.CL2026
SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge
Lucio M. Dery, Benedict Aaron Tjandra, Siavash Samiei +4
Agentic systems driven by large language models (LLMs) regularly feature two key mechanisms to autonomously solve complex problems: synthesizing text-based knowledge and procedures…
cs.LG2025
Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
Zachary Charles, Gabriel Teston, Lucio Dery +5
As we scale to more massive machine learning models, the frequent synchronization demands inherent in data-parallel approaches create significant slowdowns, posing a critical chall…