Budgeted LoRA: Distillation as Structured Compute Allocation for Efficient Inference

Mohammed Sabry, Anya Belz

We study distillation for large language models under explicit compute constraints, with the goal of producing student models that are not only cheaper to train, but structurally e…