7 citations · 14 across the 6 of their papers we have counts for
Showing 2026Show all
2 papers · 1 filter
cs.CL2026
Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model
Mohammed Sabry, Sean Augenstein, Keith Rush +1
We ask whether language-model pre-training can be decomposed into smaller, independently trainable jobs that can later be recomposed into a coherent larger model. We introduce Mixt…
cs.LG2026
Budgeted LoRA: Distillation as Structured Compute Allocation for Efficient Inference
Mohammed Sabry, Anya Belz
We study distillation for large language models under explicit compute constraints, with the goal of producing student models that are not only cheaper to train, but structurally e…