3 papers
cs.LG2026
pscaling small models: Principled warm starts and hyperparameter transfer
Yuxin Ma, Nan Chen, Mateo DÃaz +3
Modern large-scale neural networks are often trained and released in multiple sizes to accommodate diverse inference budgets. To improve efficiency, recent work has explored model…
cs.LG2026
On Transferring Transferability: Towards a Theory for Size Generalization
Eitan Levin, Yuxin Ma, Mateo DÃaz +1
Many modern learning tasks require models that can take inputs of varying sizes. Consequently, dimension-independent architectures have been proposed for domains where the inputs a…
cs.LG2026
Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning
Nan Chen, Soledad Villar, Soufiane Hayou
Low-Rank Adaptation (LoRA) is a standard tool for parameter-efficient finetuning of large models. While it induces a small memory footprint, its training dynamics can be surprising…